Discussion:
[Xquartz-dev] XQuartz 2.7.10_rc1
Jeremy Huddleston Sequoia
2016-09-09 05:06:25 UTC
Permalink
Hey all,

I'd like to announce that XQuartz 2.7.10_rc1 is available. Full changes since 2.7.9 can be seen in the release notes:

https://www.xquartz.org/releases/XQuartz-2.7.10_rc1.html

The main change since 2.7.10_beta2 is around the libXt compatibility hack for Motif. Because of issues with applications that use both Motif and libXaw, I've abandoned the previous approach of shipping /opt/X11/lib/libXt.6.dylib as flat_namespace and /opt/X11/lib/libXt.7.dylib as the two-level namespace version. Now, /opt/X11/lib/libXt.6.dylib is two-level namespace, and /opt/X11/lib/libXt.7.dylib is provided as a symlink to it for compatibility. This means that Motif based applications will fail to launch by default unless this bug is fixed in Motif.

As a workaround, we're shipping the flat namespace dylib hack at /opt/X11/lib/flat_namespace/libXt.6.dylib. Users who require that version have two options:
1) Replace the two-level namespace version with it (sudo cp /opt/X11/lib/flat_namespace/libXt.6.dylib /opt/X11/lib/libXt.6.dylib)
2) Launch your Motif applications using DYLD_LIBRARY_PATH=/opt/X11/lib/flat_namespace

This should not be seen as a long-term solution. Hopefully someone will step up to fix Motif to not require this hack.

Please give this a try and comment on the effectiveness of this approach if you use Motif-based applications.

Thanks,
Jeremy
Jeremy Huddleston Sequoia
2016-09-10 17:09:03 UTC
Permalink
I built 2.7.10_rc1 against the 10.12 SDK. This has introduced some compatibility issues when running on older OS versions. I'm going to rebuild an rc2 against the 10.11 SDK and aim to have it available tomorrow.

In the mean time, please go back to 2.7.10_beta2 if you have issues.

See: https://bugs.freedesktop.org/show_bug.cgi?id=97757

Thanks,
Jeremy
Post by Jeremy Huddleston Sequoia
Hey all,
https://www.xquartz.org/releases/XQuartz-2.7.10_rc1.html
The main change since 2.7.10_beta2 is around the libXt compatibility hack for Motif. Because of issues with applications that use both Motif and libXaw, I've abandoned the previous approach of shipping /opt/X11/lib/libXt.6.dylib as flat_namespace and /opt/X11/lib/libXt.7.dylib as the two-level namespace version. Now, /opt/X11/lib/libXt.6.dylib is two-level namespace, and /opt/X11/lib/libXt.7.dylib is provided as a symlink to it for compatibility. This means that Motif based applications will fail to launch by default unless this bug is fixed in Motif.
1) Replace the two-level namespace version with it (sudo cp /opt/X11/lib/flat_namespace/libXt.6.dylib /opt/X11/lib/libXt.6.dylib)
2) Launch your Motif applications using DYLD_LIBRARY_PATH=/opt/X11/lib/flat_namespace
This should not be seen as a long-term solution. Hopefully someone will step up to fix Motif to not require this hack.
Please give this a try and comment on the effectiveness of this approach if you use Motif-based applications.
Thanks,
Jeremy
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/x11-users/jeremyhu%40freedesktop.org
Jack Howarth
2016-09-13 12:50:52 UTC
Permalink
On Fri, Sep 9, 2016 at 1:06 AM, Jeremy Huddleston Sequoia
Post by Jeremy Huddleston Sequoia
Hey all,
https://www.xquartz.org/releases/XQuartz-2.7.10_rc1.html
The main change since 2.7.10_beta2 is around the libXt compatibility hack for Motif. Because of issues with applications that use both Motif and libXaw, I've abandoned the previous approach of shipping /opt/X11/lib/libXt.6.dylib as flat_namespace and /opt/X11/lib/libXt.7.dylib as the two-level namespace version. Now, /opt/X11/lib/libXt.6.dylib is two-level namespace, and /opt/X11/lib/libXt.7.dylib is provided as a symlink to it for compatibility. This means that Motif based applications will fail to launch by default unless this bug is fixed in Motif.
1) Replace the two-level namespace version with it (sudo cp /opt/X11/lib/flat_namespace/libXt.6.dylib /opt/X11/lib/libXt.6.dylib)
2) Launch your Motif applications using DYLD_LIBRARY_PATH=/opt/X11/lib/flat_namespace
That really isn't a viable solution since using DYLD_LIBRARY_PATH will
require users to disable rootless on 10.11 and later.
Post by Jeremy Huddleston Sequoia
This should not be seen as a long-term solution. Hopefully someone will step up to fix Motif to not require this hack.
That is impossible to achieve. The underlying problem is a namespace
conflict for at least one symbol in motif. Any proper fix for this
would require motif to rename those symbols and break all legacy motif
binaries.
Post by Jeremy Huddleston Sequoia
Please give this a try and comment on the effectiveness of this approach if you use Motif-based applications.
Is the breakage in 2.7.9 *really* that bad? It seems better to revert
this change in 2.7.10 and leave things as they are in 2.7.9 relative
to the motif support. At least legacy binaries still work there and if
users rebuild rigorously against the two-level namespace
libXt.7.dylib, there shouldn't be issues with newly built binaries.
Post by Jeremy Huddleston Sequoia
Thanks,
Jeremy
_______________________________________________
Xquartz-dev mailing list
https://lists.macosforge.org/mailman/listinfo/xquartz-dev
Jeremy Huddleston Sequoia
2016-09-13 20:12:14 UTC
Permalink
Post by Jack Howarth
On Fri, Sep 9, 2016 at 1:06 AM, Jeremy Huddleston Sequoia
Post by Jeremy Huddleston Sequoia
Hey all,
https://www.xquartz.org/releases/XQuartz-2.7.10_rc1.html
The main change since 2.7.10_beta2 is around the libXt compatibility hack for Motif. Because of issues with applications that use both Motif and libXaw, I've abandoned the previous approach of shipping /opt/X11/lib/libXt.6.dylib as flat_namespace and /opt/X11/lib/libXt.7.dylib as the two-level namespace version. Now, /opt/X11/lib/libXt.6.dylib is two-level namespace, and /opt/X11/lib/libXt.7.dylib is provided as a symlink to it for compatibility. This means that Motif based applications will fail to launch by default unless this bug is fixed in Motif.
1) Replace the two-level namespace version with it (sudo cp /opt/X11/lib/flat_namespace/libXt.6.dylib /opt/X11/lib/libXt.6.dylib)
2) Launch your Motif applications using DYLD_LIBRARY_PATH=/opt/X11/lib/flat_namespace
That really isn't a viable solution since using DYLD_LIBRARY_PATH will
require users to disable rootless on 10.11 and later.
No, that's certainly not the case. DYLD_LIBRARY_PATH is ignored for system executables. There are no system executables that require Motif, so that's not relevant.
Post by Jack Howarth
Post by Jeremy Huddleston Sequoia
This should not be seen as a long-term solution. Hopefully someone will step up to fix Motif to not require this hack.
That is impossible to achieve. The underlying problem is a namespace
conflict for at least one symbol in motif. Any proper fix for this
would require motif to rename those symbols and break all legacy motif
binaries.
Nope. You can see the change that I made to libXaw and libXaw3d for this purpose. It does not break binary compatibility for those libraries.
Post by Jack Howarth
Post by Jeremy Huddleston Sequoia
Please give this a try and comment on the effectiveness of this approach if you use Motif-based applications.
Is the breakage in 2.7.9 *really* that bad?
Yes. The issue is that the structs are actually sized differently and Motif and libXaw were overrunning the allocated memory, causing memory corruption. This fixes the problem correctly for clients of libXaw and libXaw3d. Motif just needs a similar change.
Post by Jack Howarth
It seems better to revert
this change in 2.7.10 and leave things as they are in 2.7.9 relative
to the motif support. At least legacy binaries still work there and if
users rebuild rigorously against the two-level namespace
libXt.7.dylib, there shouldn't be issues with newly built binaries.
The solution in 2.7.9 was not working for legacy executables that were including both libXt.6.dylib (through libXm.dylib) and libXt.7.dylib (through libXaw.dylib or libXaw3d.dylib). An alternative would be to bump both libXt and everything else that linked against it. That falls apart quite quickly.

---

The solution as it is basically allows users to choose memory corruption and OpenMotif compatibility (by using the ones in the flat_namespace subdirectory) or security and reliability (by using what we ship by default).

--Jeremy
Jack Howarth
2016-09-14 16:51:46 UTC
Permalink
Jeremy,
There is a major thinko in how you built XQuartz 2.7.10-rc2. The files...

$ otool -L /opt/X11/lib/flat_namespace/libXt.6.dylib
/opt/X11/lib/flat_namespace/libXt.6.dylib:
/opt/X11/lib/libXt.6.dylib (compatibility version 7.0.0, current version 7.0.0)
/opt/X11/lib/libSM.6.dylib (compatibility version 7.0.0, current version 7.1.0)
/opt/X11/lib/libICE.6.dylib (compatibility version 10.0.0, current
version 10.0.0)
/opt/X11/lib/libX11.6.dylib (compatibility version 10.0.0, current
version 10.0.0)
/opt/X11/lib/libxcb.1.dylib (compatibility version 3.0.0, current version 3.0.0)
/opt/X11/lib/libXau.6.dylib (compatibility version 7.0.0, current version 7.0.0)
/opt/X11/lib/libXdmcp.6.dylib (compatibility version 7.0.0, current
version 7.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 1225.1.1)

and

$ otool -L /opt/X11/lib/flat_namespace/libXt.7.dylib
/opt/X11/lib/flat_namespace/libXt.7.dylib:
/opt/X11/lib/libXt.6.dylib (compatibility version 7.0.0, current version 7.0.0)
/opt/X11/lib/libSM.6.dylib (compatibility version 7.0.0, current version 7.1.0)
/opt/X11/lib/libICE.6.dylib (compatibility version 10.0.0, current
version 10.0.0)
/opt/X11/lib/libX11.6.dylib (compatibility version 10.0.0, current
version 10.0.0)
/opt/X11/lib/libxcb.1.dylib (compatibility version 3.0.0, current version 3.0.0)
/opt/X11/lib/libXau.6.dylib (compatibility version 7.0.0, current version 7.0.0)
/opt/X11/lib/libXdmcp.6.dylib (compatibility version 7.0.0, current
version 7.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 1225.1.1)

have the wrong install name path. So any binaries linked against those
two flat-namespace shared libraries are going to load the two-level
namespace ones instead.
Jack


On Tue, Sep 13, 2016 at 4:12 PM, Jeremy Huddleston Sequoia
Post by Jeremy Huddleston Sequoia
Post by Jack Howarth
On Fri, Sep 9, 2016 at 1:06 AM, Jeremy Huddleston Sequoia
Post by Jeremy Huddleston Sequoia
Hey all,
https://www.xquartz.org/releases/XQuartz-2.7.10_rc1.html
The main change since 2.7.10_beta2 is around the libXt compatibility hack for Motif. Because of issues with applications that use both Motif and libXaw, I've abandoned the previous approach of shipping /opt/X11/lib/libXt.6.dylib as flat_namespace and /opt/X11/lib/libXt.7.dylib as the two-level namespace version. Now, /opt/X11/lib/libXt.6.dylib is two-level namespace, and /opt/X11/lib/libXt.7.dylib is provided as a symlink to it for compatibility. This means that Motif based applications will fail to launch by default unless this bug is fixed in Motif.
1) Replace the two-level namespace version with it (sudo cp /opt/X11/lib/flat_namespace/libXt.6.dylib /opt/X11/lib/libXt.6.dylib)
2) Launch your Motif applications using DYLD_LIBRARY_PATH=/opt/X11/lib/flat_namespace
That really isn't a viable solution since using DYLD_LIBRARY_PATH will
require users to disable rootless on 10.11 and later.
No, that's certainly not the case. DYLD_LIBRARY_PATH is ignored for system executables. There are no system executables that require Motif, so that's not relevant.
Post by Jack Howarth
Post by Jeremy Huddleston Sequoia
This should not be seen as a long-term solution. Hopefully someone will step up to fix Motif to not require this hack.
That is impossible to achieve. The underlying problem is a namespace
conflict for at least one symbol in motif. Any proper fix for this
would require motif to rename those symbols and break all legacy motif
binaries.
Nope. You can see the change that I made to libXaw and libXaw3d for this purpose. It does not break binary compatibility for those libraries.
Post by Jack Howarth
Post by Jeremy Huddleston Sequoia
Please give this a try and comment on the effectiveness of this approach if you use Motif-based applications.
Is the breakage in 2.7.9 *really* that bad?
Yes. The issue is that the structs are actually sized differently and Motif and libXaw were overrunning the allocated memory, causing memory corruption. This fixes the problem correctly for clients of libXaw and libXaw3d. Motif just needs a similar change.
Post by Jack Howarth
It seems better to revert
this change in 2.7.10 and leave things as they are in 2.7.9 relative
to the motif support. At least legacy binaries still work there and if
users rebuild rigorously against the two-level namespace
libXt.7.dylib, there shouldn't be issues with newly built binaries.
The solution in 2.7.9 was not working for legacy executables that were including both libXt.6.dylib (through libXm.dylib) and libXt.7.dylib (through libXaw.dylib or libXaw3d.dylib). An alternative would be to bump both libXt and everything else that linked against it. That falls apart quite quickly.
---
The solution as it is basically allows users to choose memory corruption and OpenMotif compatibility (by using the ones in the flat_namespace subdirectory) or security and reliability (by using what we ship by default).
--Jeremy
_______________________________________________
Xquartz-dev mailing list
https://lists.macosforge.org/mailman/listinfo/xquartz-dev
Jeremy Huddleston Sequoia
2016-09-14 17:53:37 UTC
Permalink
Post by Jack Howarth
Jeremy,
There is a major thinko in how you built XQuartz 2.7.10-rc2. The files...
$ otool -L /opt/X11/lib/flat_namespace/libXt.6.dylib
/opt/X11/lib/libXt.6.dylib (compatibility version 7.0.0, current version 7.0.0)
/opt/X11/lib/libSM.6.dylib (compatibility version 7.0.0, current version 7.1.0)
/opt/X11/lib/libICE.6.dylib (compatibility version 10.0.0, current
version 10.0.0)
/opt/X11/lib/libX11.6.dylib (compatibility version 10.0.0, current
version 10.0.0)
/opt/X11/lib/libxcb.1.dylib (compatibility version 3.0.0, current version 3.0.0)
/opt/X11/lib/libXau.6.dylib (compatibility version 7.0.0, current version 7.0.0)
/opt/X11/lib/libXdmcp.6.dylib (compatibility version 7.0.0, current
version 7.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 1225.1.1)
and
$ otool -L /opt/X11/lib/flat_namespace/libXt.7.dylib
/opt/X11/lib/libXt.6.dylib (compatibility version 7.0.0, current version 7.0.0)
/opt/X11/lib/libSM.6.dylib (compatibility version 7.0.0, current version 7.1.0)
/opt/X11/lib/libICE.6.dylib (compatibility version 10.0.0, current
version 10.0.0)
/opt/X11/lib/libX11.6.dylib (compatibility version 10.0.0, current
version 10.0.0)
/opt/X11/lib/libxcb.1.dylib (compatibility version 3.0.0, current version 3.0.0)
/opt/X11/lib/libXau.6.dylib (compatibility version 7.0.0, current version 7.0.0)
/opt/X11/lib/libXdmcp.6.dylib (compatibility version 7.0.0, current
version 7.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 1225.1.1)
have the wrong install name path. So any binaries linked against those
two flat-namespace shared libraries are going to load the two-level
namespace ones instead.
Nope, that is exactly correct. Projects should NOT be linking against those libraries. They are being shipped for runtime usage only. Parties that are interested in using those instead of the ones we ship by default can copy them into location or use DYLD_LIBRARY_PATH as suggested earlier.

Any developer trying to link against those binaries should instead invest time in fixing Motif. There's no reason anyone should be linking against those directly.

As an alternative, I could just ship a separate Motif compatibility package which lays those down in /opt/X11/lib, but I fear that doing so might cause people to install it that don't really need it.
Post by Jack Howarth
Jack
On Tue, Sep 13, 2016 at 4:12 PM, Jeremy Huddleston Sequoia
Post by Jeremy Huddleston Sequoia
Post by Jack Howarth
On Fri, Sep 9, 2016 at 1:06 AM, Jeremy Huddleston Sequoia
Post by Jeremy Huddleston Sequoia
Hey all,
https://www.xquartz.org/releases/XQuartz-2.7.10_rc1.html
The main change since 2.7.10_beta2 is around the libXt compatibility hack for Motif. Because of issues with applications that use both Motif and libXaw, I've abandoned the previous approach of shipping /opt/X11/lib/libXt.6.dylib as flat_namespace and /opt/X11/lib/libXt.7.dylib as the two-level namespace version. Now, /opt/X11/lib/libXt.6.dylib is two-level namespace, and /opt/X11/lib/libXt.7.dylib is provided as a symlink to it for compatibility. This means that Motif based applications will fail to launch by default unless this bug is fixed in Motif.
1) Replace the two-level namespace version with it (sudo cp /opt/X11/lib/flat_namespace/libXt.6.dylib /opt/X11/lib/libXt.6.dylib)
2) Launch your Motif applications using DYLD_LIBRARY_PATH=/opt/X11/lib/flat_namespace
That really isn't a viable solution since using DYLD_LIBRARY_PATH will
require users to disable rootless on 10.11 and later.
No, that's certainly not the case. DYLD_LIBRARY_PATH is ignored for system executables. There are no system executables that require Motif, so that's not relevant.
Post by Jack Howarth
Post by Jeremy Huddleston Sequoia
This should not be seen as a long-term solution. Hopefully someone will step up to fix Motif to not require this hack.
That is impossible to achieve. The underlying problem is a namespace
conflict for at least one symbol in motif. Any proper fix for this
would require motif to rename those symbols and break all legacy motif
binaries.
Nope. You can see the change that I made to libXaw and libXaw3d for this purpose. It does not break binary compatibility for those libraries.
Post by Jack Howarth
Post by Jeremy Huddleston Sequoia
Please give this a try and comment on the effectiveness of this approach if you use Motif-based applications.
Is the breakage in 2.7.9 *really* that bad?
Yes. The issue is that the structs are actually sized differently and Motif and libXaw were overrunning the allocated memory, causing memory corruption. This fixes the problem correctly for clients of libXaw and libXaw3d. Motif just needs a similar change.
Post by Jack Howarth
It seems better to revert
this change in 2.7.10 and leave things as they are in 2.7.9 relative
to the motif support. At least legacy binaries still work there and if
users rebuild rigorously against the two-level namespace
libXt.7.dylib, there shouldn't be issues with newly built binaries.
The solution in 2.7.9 was not working for legacy executables that were including both libXt.6.dylib (through libXm.dylib) and libXt.7.dylib (through libXaw.dylib or libXaw3d.dylib). An alternative would be to bump both libXt and everything else that linked against it. That falls apart quite quickly.
---
The solution as it is basically allows users to choose memory corruption and OpenMotif compatibility (by using the ones in the flat_namespace subdirectory) or security and reliability (by using what we ship by default).
--Jeremy
_______________________________________________
Xquartz-dev mailing list
https://lists.macosforge.org/mailman/listinfo/xquartz-dev
_______________________________________________
Xquartz-dev mailing list
https://lists.macosforge.org/mailman/listinfo/xquartz-dev
Jack Howarth
2016-09-14 19:23:25 UTC
Permalink
On Wed, Sep 14, 2016 at 1:53 PM, Jeremy Huddleston Sequoia
Post by Jeremy Huddleston Sequoia
Post by Jack Howarth
Jeremy,
There is a major thinko in how you built XQuartz 2.7.10-rc2. The files...
$ otool -L /opt/X11/lib/flat_namespace/libXt.6.dylib
/opt/X11/lib/libXt.6.dylib (compatibility version 7.0.0, current version 7.0.0)
/opt/X11/lib/libSM.6.dylib (compatibility version 7.0.0, current version 7.1.0)
/opt/X11/lib/libICE.6.dylib (compatibility version 10.0.0, current
version 10.0.0)
/opt/X11/lib/libX11.6.dylib (compatibility version 10.0.0, current
version 10.0.0)
/opt/X11/lib/libxcb.1.dylib (compatibility version 3.0.0, current version 3.0.0)
/opt/X11/lib/libXau.6.dylib (compatibility version 7.0.0, current version 7.0.0)
/opt/X11/lib/libXdmcp.6.dylib (compatibility version 7.0.0, current
version 7.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 1225.1.1)
and
$ otool -L /opt/X11/lib/flat_namespace/libXt.7.dylib
/opt/X11/lib/libXt.6.dylib (compatibility version 7.0.0, current version 7.0.0)
/opt/X11/lib/libSM.6.dylib (compatibility version 7.0.0, current version 7.1.0)
/opt/X11/lib/libICE.6.dylib (compatibility version 10.0.0, current
version 10.0.0)
/opt/X11/lib/libX11.6.dylib (compatibility version 10.0.0, current
version 10.0.0)
/opt/X11/lib/libxcb.1.dylib (compatibility version 3.0.0, current version 3.0.0)
/opt/X11/lib/libXau.6.dylib (compatibility version 7.0.0, current version 7.0.0)
/opt/X11/lib/libXdmcp.6.dylib (compatibility version 7.0.0, current
version 7.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 1225.1.1)
have the wrong install name path. So any binaries linked against those
two flat-namespace shared libraries are going to load the two-level
namespace ones instead.
Nope, that is exactly correct. Projects should NOT be linking against those libraries. They are being shipped for runtime usage only. Parties that are interested in using those instead of the ones we ship by default can copy them into location or use DYLD_LIBRARY_PATH as suggested earlier.
Any developer trying to link against those binaries should instead invest time in fixing Motif. There's no reason anyone should be linking against those directly.
I am pretty certain that there is no way to fix this that doesn't
break the motif ABI with pre-existing binaries because the crux of the
problem is a name-space collision with libXt. My understanding was
that linux doesn't tickle this problem because their weak symbol
handling allows the overlapping symbols from motif to dominate over
those from libXt.
Post by Jeremy Huddleston Sequoia
As an alternative, I could just ship a separate Motif compatibility package which lays those down in /opt/X11/lib, but I fear that doing so might cause people to install it that don't really need it.
Post by Jack Howarth
Jack
On Tue, Sep 13, 2016 at 4:12 PM, Jeremy Huddleston Sequoia
Post by Jeremy Huddleston Sequoia
Post by Jack Howarth
On Fri, Sep 9, 2016 at 1:06 AM, Jeremy Huddleston Sequoia
Post by Jeremy Huddleston Sequoia
Hey all,
https://www.xquartz.org/releases/XQuartz-2.7.10_rc1.html
The main change since 2.7.10_beta2 is around the libXt compatibility hack for Motif. Because of issues with applications that use both Motif and libXaw, I've abandoned the previous approach of shipping /opt/X11/lib/libXt.6.dylib as flat_namespace and /opt/X11/lib/libXt.7.dylib as the two-level namespace version. Now, /opt/X11/lib/libXt.6.dylib is two-level namespace, and /opt/X11/lib/libXt.7.dylib is provided as a symlink to it for compatibility. This means that Motif based applications will fail to launch by default unless this bug is fixed in Motif.
1) Replace the two-level namespace version with it (sudo cp /opt/X11/lib/flat_namespace/libXt.6.dylib /opt/X11/lib/libXt.6.dylib)
2) Launch your Motif applications using DYLD_LIBRARY_PATH=/opt/X11/lib/flat_namespace
That really isn't a viable solution since using DYLD_LIBRARY_PATH will
require users to disable rootless on 10.11 and later.
No, that's certainly not the case. DYLD_LIBRARY_PATH is ignored for system executables. There are no system executables that require Motif, so that's not relevant.
Post by Jack Howarth
Post by Jeremy Huddleston Sequoia
This should not be seen as a long-term solution. Hopefully someone will step up to fix Motif to not require this hack.
That is impossible to achieve. The underlying problem is a namespace
conflict for at least one symbol in motif. Any proper fix for this
would require motif to rename those symbols and break all legacy motif
binaries.
Nope. You can see the change that I made to libXaw and libXaw3d for this purpose. It does not break binary compatibility for those libraries.
Post by Jack Howarth
Post by Jeremy Huddleston Sequoia
Please give this a try and comment on the effectiveness of this approach if you use Motif-based applications.
Is the breakage in 2.7.9 *really* that bad?
Yes. The issue is that the structs are actually sized differently and Motif and libXaw were overrunning the allocated memory, causing memory corruption. This fixes the problem correctly for clients of libXaw and libXaw3d. Motif just needs a similar change.
Post by Jack Howarth
It seems better to revert
this change in 2.7.10 and leave things as they are in 2.7.9 relative
to the motif support. At least legacy binaries still work there and if
users rebuild rigorously against the two-level namespace
libXt.7.dylib, there shouldn't be issues with newly built binaries.
The solution in 2.7.9 was not working for legacy executables that were including both libXt.6.dylib (through libXm.dylib) and libXt.7.dylib (through libXaw.dylib or libXaw3d.dylib). An alternative would be to bump both libXt and everything else that linked against it. That falls apart quite quickly.
---
The solution as it is basically allows users to choose memory corruption and OpenMotif compatibility (by using the ones in the flat_namespace subdirectory) or security and reliability (by using what we ship by default).
--Jeremy
_______________________________________________
Xquartz-dev mailing list
https://lists.macosforge.org/mailman/listinfo/xquartz-dev
_______________________________________________
Xquartz-dev mailing list
https://lists.macosforge.org/mailman/listinfo/xquartz-dev
_______________________________________________
Xquartz-dev mailing list
https://lists.macosforge.org/mailman/listinfo/xquartz-dev
Jack Howarth
2016-09-14 21:15:01 UTC
Permalink
On Wed, Sep 14, 2016 at 3:23 PM, Jack Howarth
Post by Jack Howarth
On Wed, Sep 14, 2016 at 1:53 PM, Jeremy Huddleston Sequoia
Post by Jeremy Huddleston Sequoia
Post by Jack Howarth
Jeremy,
There is a major thinko in how you built XQuartz 2.7.10-rc2. The files...
$ otool -L /opt/X11/lib/flat_namespace/libXt.6.dylib
/opt/X11/lib/libXt.6.dylib (compatibility version 7.0.0, current version 7.0.0)
/opt/X11/lib/libSM.6.dylib (compatibility version 7.0.0, current version 7.1.0)
/opt/X11/lib/libICE.6.dylib (compatibility version 10.0.0, current
version 10.0.0)
/opt/X11/lib/libX11.6.dylib (compatibility version 10.0.0, current
version 10.0.0)
/opt/X11/lib/libxcb.1.dylib (compatibility version 3.0.0, current version 3.0.0)
/opt/X11/lib/libXau.6.dylib (compatibility version 7.0.0, current version 7.0.0)
/opt/X11/lib/libXdmcp.6.dylib (compatibility version 7.0.0, current
version 7.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 1225.1.1)
and
$ otool -L /opt/X11/lib/flat_namespace/libXt.7.dylib
/opt/X11/lib/libXt.6.dylib (compatibility version 7.0.0, current version 7.0.0)
/opt/X11/lib/libSM.6.dylib (compatibility version 7.0.0, current version 7.1.0)
/opt/X11/lib/libICE.6.dylib (compatibility version 10.0.0, current
version 10.0.0)
/opt/X11/lib/libX11.6.dylib (compatibility version 10.0.0, current
version 10.0.0)
/opt/X11/lib/libxcb.1.dylib (compatibility version 3.0.0, current version 3.0.0)
/opt/X11/lib/libXau.6.dylib (compatibility version 7.0.0, current version 7.0.0)
/opt/X11/lib/libXdmcp.6.dylib (compatibility version 7.0.0, current
version 7.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 1225.1.1)
have the wrong install name path. So any binaries linked against those
two flat-namespace shared libraries are going to load the two-level
namespace ones instead.
Nope, that is exactly correct. Projects should NOT be linking against those libraries. They are being shipped for runtime usage only. Parties that are interested in using those instead of the ones we ship by default can copy them into location or use DYLD_LIBRARY_PATH as suggested earlier.
Any developer trying to link against those binaries should instead invest time in fixing Motif. There's no reason anyone should be linking against those directly.
I am pretty certain that there is no way to fix this that doesn't
break the motif ABI with pre-existing binaries because the crux of the
problem is a name-space collision with libXt. My understanding was
that linux doesn't tickle this problem because their weak symbol
handling allows the overlapping symbols from motif to dominate over
those from libXt.
One thing that might be worth trying (which I haven't puzzled out yet)
is changing the symbols...

# nm libXm.4.dylib | grep vendorShell
00000000002023f0 D _vendorShellClassRec
0000000000202510 D _vendorShellWidgetClass

from 'D' to 'd' with the appropriate invocation of
__attribute__((weak_import)). So far I haven't found the proper place
to insert that to get the symbol converted into a weak_import.
Post by Jack Howarth
Post by Jeremy Huddleston Sequoia
As an alternative, I could just ship a separate Motif compatibility package which lays those down in /opt/X11/lib, but I fear that doing so might cause people to install it that don't really need it.
Post by Jack Howarth
Jack
On Tue, Sep 13, 2016 at 4:12 PM, Jeremy Huddleston Sequoia
Post by Jeremy Huddleston Sequoia
Post by Jack Howarth
On Fri, Sep 9, 2016 at 1:06 AM, Jeremy Huddleston Sequoia
Post by Jeremy Huddleston Sequoia
Hey all,
https://www.xquartz.org/releases/XQuartz-2.7.10_rc1.html
The main change since 2.7.10_beta2 is around the libXt compatibility hack for Motif. Because of issues with applications that use both Motif and libXaw, I've abandoned the previous approach of shipping /opt/X11/lib/libXt.6.dylib as flat_namespace and /opt/X11/lib/libXt.7.dylib as the two-level namespace version. Now, /opt/X11/lib/libXt.6.dylib is two-level namespace, and /opt/X11/lib/libXt.7.dylib is provided as a symlink to it for compatibility. This means that Motif based applications will fail to launch by default unless this bug is fixed in Motif.
1) Replace the two-level namespace version with it (sudo cp /opt/X11/lib/flat_namespace/libXt.6.dylib /opt/X11/lib/libXt.6.dylib)
2) Launch your Motif applications using DYLD_LIBRARY_PATH=/opt/X11/lib/flat_namespace
That really isn't a viable solution since using DYLD_LIBRARY_PATH will
require users to disable rootless on 10.11 and later.
No, that's certainly not the case. DYLD_LIBRARY_PATH is ignored for system executables. There are no system executables that require Motif, so that's not relevant.
Post by Jack Howarth
Post by Jeremy Huddleston Sequoia
This should not be seen as a long-term solution. Hopefully someone will step up to fix Motif to not require this hack.
That is impossible to achieve. The underlying problem is a namespace
conflict for at least one symbol in motif. Any proper fix for this
would require motif to rename those symbols and break all legacy motif
binaries.
Nope. You can see the change that I made to libXaw and libXaw3d for this purpose. It does not break binary compatibility for those libraries.
Post by Jack Howarth
Post by Jeremy Huddleston Sequoia
Please give this a try and comment on the effectiveness of this approach if you use Motif-based applications.
Is the breakage in 2.7.9 *really* that bad?
Yes. The issue is that the structs are actually sized differently and Motif and libXaw were overrunning the allocated memory, causing memory corruption. This fixes the problem correctly for clients of libXaw and libXaw3d. Motif just needs a similar change.
Post by Jack Howarth
It seems better to revert
this change in 2.7.10 and leave things as they are in 2.7.9 relative
to the motif support. At least legacy binaries still work there and if
users rebuild rigorously against the two-level namespace
libXt.7.dylib, there shouldn't be issues with newly built binaries.
The solution in 2.7.9 was not working for legacy executables that were including both libXt.6.dylib (through libXm.dylib) and libXt.7.dylib (through libXaw.dylib or libXaw3d.dylib). An alternative would be to bump both libXt and everything else that linked against it. That falls apart quite quickly.
---
The solution as it is basically allows users to choose memory corruption and OpenMotif compatibility (by using the ones in the flat_namespace subdirectory) or security and reliability (by using what we ship by default).
--Jeremy
_______________________________________________
Xquartz-dev mailing list
https://lists.macosforge.org/mailman/listinfo/xquartz-dev
_______________________________________________
Xquartz-dev mailing list
https://lists.macosforge.org/mailman/listinfo/xquartz-dev
_______________________________________________
Xquartz-dev mailing list
https://lists.macosforge.org/mailman/listinfo/xquartz-dev
Jack Howarth
2016-09-14 21:41:39 UTC
Permalink
On Wed, Sep 14, 2016 at 5:15 PM, Jack Howarth
Post by Jack Howarth
On Wed, Sep 14, 2016 at 3:23 PM, Jack Howarth
Post by Jack Howarth
On Wed, Sep 14, 2016 at 1:53 PM, Jeremy Huddleston Sequoia
Post by Jeremy Huddleston Sequoia
Post by Jack Howarth
Jeremy,
There is a major thinko in how you built XQuartz 2.7.10-rc2. The files...
$ otool -L /opt/X11/lib/flat_namespace/libXt.6.dylib
/opt/X11/lib/libXt.6.dylib (compatibility version 7.0.0, current version 7.0.0)
/opt/X11/lib/libSM.6.dylib (compatibility version 7.0.0, current version 7.1.0)
/opt/X11/lib/libICE.6.dylib (compatibility version 10.0.0, current
version 10.0.0)
/opt/X11/lib/libX11.6.dylib (compatibility version 10.0.0, current
version 10.0.0)
/opt/X11/lib/libxcb.1.dylib (compatibility version 3.0.0, current version 3.0.0)
/opt/X11/lib/libXau.6.dylib (compatibility version 7.0.0, current version 7.0.0)
/opt/X11/lib/libXdmcp.6.dylib (compatibility version 7.0.0, current
version 7.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 1225.1.1)
and
$ otool -L /opt/X11/lib/flat_namespace/libXt.7.dylib
/opt/X11/lib/libXt.6.dylib (compatibility version 7.0.0, current version 7.0.0)
/opt/X11/lib/libSM.6.dylib (compatibility version 7.0.0, current version 7.1.0)
/opt/X11/lib/libICE.6.dylib (compatibility version 10.0.0, current
version 10.0.0)
/opt/X11/lib/libX11.6.dylib (compatibility version 10.0.0, current
version 10.0.0)
/opt/X11/lib/libxcb.1.dylib (compatibility version 3.0.0, current version 3.0.0)
/opt/X11/lib/libXau.6.dylib (compatibility version 7.0.0, current version 7.0.0)
/opt/X11/lib/libXdmcp.6.dylib (compatibility version 7.0.0, current
version 7.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 1225.1.1)
have the wrong install name path. So any binaries linked against those
two flat-namespace shared libraries are going to load the two-level
namespace ones instead.
Nope, that is exactly correct. Projects should NOT be linking against those libraries. They are being shipped for runtime usage only. Parties that are interested in using those instead of the ones we ship by default can copy them into location or use DYLD_LIBRARY_PATH as suggested earlier.
Any developer trying to link against those binaries should instead invest time in fixing Motif. There's no reason anyone should be linking against those directly.
I am pretty certain that there is no way to fix this that doesn't
break the motif ABI with pre-existing binaries because the crux of the
problem is a name-space collision with libXt. My understanding was
that linux doesn't tickle this problem because their weak symbol
handling allows the overlapping symbols from motif to dominate over
those from libXt.
One thing that might be worth trying (which I haven't puzzled out yet)
is changing the symbols...
# nm libXm.4.dylib | grep vendorShell
00000000002023f0 D _vendorShellClassRec
0000000000202510 D _vendorShellWidgetClass
from 'D' to 'd' with the appropriate invocation of
__attribute__((weak_import)). So far I haven't found the proper place
to insert that to get the symbol converted into a weak_import.
Looking at this again, I am pretty sure the problem is the name space
conflict with vendorShellClassRec and vendorShellWidgetClass between
libXt and the redefined ones in libXm. Both symbols are redefined in
VendorS.c but also used in DragOverS.c and GrabShell.c for
VendorShellClassRec and in BaseClass.c and GrabShell.c for
VendorShellClassRec. The flat_namespace linkage of libXt suppressed
the mixing of the redefined symbols.
Post by Jack Howarth
Post by Jack Howarth
Post by Jeremy Huddleston Sequoia
As an alternative, I could just ship a separate Motif compatibility package which lays those down in /opt/X11/lib, but I fear that doing so might cause people to install it that don't really need it.
Post by Jack Howarth
Jack
On Tue, Sep 13, 2016 at 4:12 PM, Jeremy Huddleston Sequoia
Post by Jeremy Huddleston Sequoia
Post by Jack Howarth
On Fri, Sep 9, 2016 at 1:06 AM, Jeremy Huddleston Sequoia
Post by Jeremy Huddleston Sequoia
Hey all,
https://www.xquartz.org/releases/XQuartz-2.7.10_rc1.html
The main change since 2.7.10_beta2 is around the libXt compatibility hack for Motif. Because of issues with applications that use both Motif and libXaw, I've abandoned the previous approach of shipping /opt/X11/lib/libXt.6.dylib as flat_namespace and /opt/X11/lib/libXt.7.dylib as the two-level namespace version. Now, /opt/X11/lib/libXt.6.dylib is two-level namespace, and /opt/X11/lib/libXt.7.dylib is provided as a symlink to it for compatibility. This means that Motif based applications will fail to launch by default unless this bug is fixed in Motif.
1) Replace the two-level namespace version with it (sudo cp /opt/X11/lib/flat_namespace/libXt.6.dylib /opt/X11/lib/libXt.6.dylib)
2) Launch your Motif applications using DYLD_LIBRARY_PATH=/opt/X11/lib/flat_namespace
That really isn't a viable solution since using DYLD_LIBRARY_PATH will
require users to disable rootless on 10.11 and later.
No, that's certainly not the case. DYLD_LIBRARY_PATH is ignored for system executables. There are no system executables that require Motif, so that's not relevant.
Post by Jack Howarth
Post by Jeremy Huddleston Sequoia
This should not be seen as a long-term solution. Hopefully someone will step up to fix Motif to not require this hack.
That is impossible to achieve. The underlying problem is a namespace
conflict for at least one symbol in motif. Any proper fix for this
would require motif to rename those symbols and break all legacy motif
binaries.
Nope. You can see the change that I made to libXaw and libXaw3d for this purpose. It does not break binary compatibility for those libraries.
Post by Jack Howarth
Post by Jeremy Huddleston Sequoia
Please give this a try and comment on the effectiveness of this approach if you use Motif-based applications.
Is the breakage in 2.7.9 *really* that bad?
Yes. The issue is that the structs are actually sized differently and Motif and libXaw were overrunning the allocated memory, causing memory corruption. This fixes the problem correctly for clients of libXaw and libXaw3d. Motif just needs a similar change.
Post by Jack Howarth
It seems better to revert
this change in 2.7.10 and leave things as they are in 2.7.9 relative
to the motif support. At least legacy binaries still work there and if
users rebuild rigorously against the two-level namespace
libXt.7.dylib, there shouldn't be issues with newly built binaries.
The solution in 2.7.9 was not working for legacy executables that were including both libXt.6.dylib (through libXm.dylib) and libXt.7.dylib (through libXaw.dylib or libXaw3d.dylib). An alternative would be to bump both libXt and everything else that linked against it. That falls apart quite quickly.
---
The solution as it is basically allows users to choose memory corruption and OpenMotif compatibility (by using the ones in the flat_namespace subdirectory) or security and reliability (by using what we ship by default).
--Jeremy
_______________________________________________
Xquartz-dev mailing list
https://lists.macosforge.org/mailman/listinfo/xquartz-dev
_______________________________________________
Xquartz-dev mailing list
https://lists.macosforge.org/mailman/listinfo/xquartz-dev
_______________________________________________
Xquartz-dev mailing list
https://lists.macosforge.org/mailman/listinfo/xquartz-dev
Jeremy Huddleston Sequoia
2016-09-14 21:53:42 UTC
Permalink
Post by Jack Howarth
On Wed, Sep 14, 2016 at 5:15 PM, Jack Howarth
Post by Jack Howarth
On Wed, Sep 14, 2016 at 3:23 PM, Jack Howarth
Post by Jack Howarth
On Wed, Sep 14, 2016 at 1:53 PM, Jeremy Huddleston Sequoia
Post by Jeremy Huddleston Sequoia
Post by Jack Howarth
Jeremy,
There is a major thinko in how you built XQuartz 2.7.10-rc2. The files...
$ otool -L /opt/X11/lib/flat_namespace/libXt.6.dylib
/opt/X11/lib/libXt.6.dylib (compatibility version 7.0.0, current version 7.0.0)
/opt/X11/lib/libSM.6.dylib (compatibility version 7.0.0, current version 7.1.0)
/opt/X11/lib/libICE.6.dylib (compatibility version 10.0.0, current
version 10.0.0)
/opt/X11/lib/libX11.6.dylib (compatibility version 10.0.0, current
version 10.0.0)
/opt/X11/lib/libxcb.1.dylib (compatibility version 3.0.0, current version 3.0.0)
/opt/X11/lib/libXau.6.dylib (compatibility version 7.0.0, current version 7.0.0)
/opt/X11/lib/libXdmcp.6.dylib (compatibility version 7.0.0, current
version 7.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 1225.1.1)
and
$ otool -L /opt/X11/lib/flat_namespace/libXt.7.dylib
/opt/X11/lib/libXt.6.dylib (compatibility version 7.0.0, current version 7.0.0)
/opt/X11/lib/libSM.6.dylib (compatibility version 7.0.0, current version 7.1.0)
/opt/X11/lib/libICE.6.dylib (compatibility version 10.0.0, current
version 10.0.0)
/opt/X11/lib/libX11.6.dylib (compatibility version 10.0.0, current
version 10.0.0)
/opt/X11/lib/libxcb.1.dylib (compatibility version 3.0.0, current version 3.0.0)
/opt/X11/lib/libXau.6.dylib (compatibility version 7.0.0, current version 7.0.0)
/opt/X11/lib/libXdmcp.6.dylib (compatibility version 7.0.0, current
version 7.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 1225.1.1)
have the wrong install name path. So any binaries linked against those
two flat-namespace shared libraries are going to load the two-level
namespace ones instead.
Nope, that is exactly correct. Projects should NOT be linking against those libraries. They are being shipped for runtime usage only. Parties that are interested in using those instead of the ones we ship by default can copy them into location or use DYLD_LIBRARY_PATH as suggested earlier.
Any developer trying to link against those binaries should instead invest time in fixing Motif. There's no reason anyone should be linking against those directly.
I am pretty certain that there is no way to fix this that doesn't
break the motif ABI with pre-existing binaries because the crux of the
problem is a name-space collision with libXt. My understanding was
that linux doesn't tickle this problem because their weak symbol
handling allows the overlapping symbols from motif to dominate over
those from libXt.
One thing that might be worth trying (which I haven't puzzled out yet)
is changing the symbols...
# nm libXm.4.dylib | grep vendorShell
00000000002023f0 D _vendorShellClassRec
0000000000202510 D _vendorShellWidgetClass
from 'D' to 'd' with the appropriate invocation of
__attribute__((weak_import)). So far I haven't found the proper place
to insert that to get the symbol converted into a weak_import.
Looking at this again, I am pretty sure the problem is the name space
conflict with vendorShellClassRec and vendorShellWidgetClass between
libXt and the redefined ones in libXm. Both symbols are redefined in
VendorS.c but also used in DragOverS.c and GrabShell.c for
VendorShellClassRec and in BaseClass.c and GrabShell.c for
VendorShellClassRec. The flat_namespace linkage of libXt suppressed
the mixing of the redefined symbols.
Yep. That's what we discussed in IRC earlier this year (https://echelog.com/logs/browse/macports/1452121200). That was the same issue with libXaw, which was fixed by not exporting those symbols and wiring up libXt's vendorShellWidgetClass for Athena:

https://cgit.freedesktop.org/xorg/lib/libXaw/commit/?id=b3049d9b13333c0e67f1f23959227020741f486b

Making those symbols weak won't quite work because then Motif would still be setting the local variable's value instead of the one in libXt.
Post by Jack Howarth
Post by Jack Howarth
Post by Jack Howarth
Post by Jeremy Huddleston Sequoia
As an alternative, I could just ship a separate Motif compatibility package which lays those down in /opt/X11/lib, but I fear that doing so might cause people to install it that don't really need it.
Post by Jack Howarth
Jack
On Tue, Sep 13, 2016 at 4:12 PM, Jeremy Huddleston Sequoia
Post by Jeremy Huddleston Sequoia
Post by Jack Howarth
On Fri, Sep 9, 2016 at 1:06 AM, Jeremy Huddleston Sequoia
Post by Jeremy Huddleston Sequoia
Hey all,
https://www.xquartz.org/releases/XQuartz-2.7.10_rc1.html
The main change since 2.7.10_beta2 is around the libXt compatibility hack for Motif. Because of issues with applications that use both Motif and libXaw, I've abandoned the previous approach of shipping /opt/X11/lib/libXt.6.dylib as flat_namespace and /opt/X11/lib/libXt.7.dylib as the two-level namespace version. Now, /opt/X11/lib/libXt.6.dylib is two-level namespace, and /opt/X11/lib/libXt.7.dylib is provided as a symlink to it for compatibility. This means that Motif based applications will fail to launch by default unless this bug is fixed in Motif.
1) Replace the two-level namespace version with it (sudo cp /opt/X11/lib/flat_namespace/libXt.6.dylib /opt/X11/lib/libXt.6.dylib)
2) Launch your Motif applications using DYLD_LIBRARY_PATH=/opt/X11/lib/flat_namespace
That really isn't a viable solution since using DYLD_LIBRARY_PATH will
require users to disable rootless on 10.11 and later.
No, that's certainly not the case. DYLD_LIBRARY_PATH is ignored for system executables. There are no system executables that require Motif, so that's not relevant.
Post by Jack Howarth
Post by Jeremy Huddleston Sequoia
This should not be seen as a long-term solution. Hopefully someone will step up to fix Motif to not require this hack.
That is impossible to achieve. The underlying problem is a namespace
conflict for at least one symbol in motif. Any proper fix for this
would require motif to rename those symbols and break all legacy motif
binaries.
Nope. You can see the change that I made to libXaw and libXaw3d for this purpose. It does not break binary compatibility for those libraries.
Post by Jack Howarth
Post by Jeremy Huddleston Sequoia
Please give this a try and comment on the effectiveness of this approach if you use Motif-based applications.
Is the breakage in 2.7.9 *really* that bad?
Yes. The issue is that the structs are actually sized differently and Motif and libXaw were overrunning the allocated memory, causing memory corruption. This fixes the problem correctly for clients of libXaw and libXaw3d. Motif just needs a similar change.
Post by Jack Howarth
It seems better to revert
this change in 2.7.10 and leave things as they are in 2.7.9 relative
to the motif support. At least legacy binaries still work there and if
users rebuild rigorously against the two-level namespace
libXt.7.dylib, there shouldn't be issues with newly built binaries.
The solution in 2.7.9 was not working for legacy executables that were including both libXt.6.dylib (through libXm.dylib) and libXt.7.dylib (through libXaw.dylib or libXaw3d.dylib). An alternative would be to bump both libXt and everything else that linked against it. That falls apart quite quickly.
---
The solution as it is basically allows users to choose memory corruption and OpenMotif compatibility (by using the ones in the flat_namespace subdirectory) or security and reliability (by using what we ship by default).
--Jeremy
_______________________________________________
Xquartz-dev mailing list
https://lists.macosforge.org/mailman/listinfo/xquartz-dev
_______________________________________________
Xquartz-dev mailing list
https://lists.macosforge.org/mailman/listinfo/xquartz-dev
_______________________________________________
Xquartz-dev mailing list
https://lists.macosforge.org/mailman/listinfo/xquartz-dev
_______________________________________________
Xquartz-dev mailing list
https://lists.macosforge.org/mailman/listinfo/xquartz-dev
Loading...