Freeswitch on ARM, Memory alignment issues
Summary: Has anyone looked at/actively looking at the 32-bit boundary memory alignment requirement for the ARM architecture? Detail: I am running freeswitch on an ARM platform (dockstar based on Marvell Kirkwood). I have compiled and tested both freeswitch-1.0.6 and freeswitch-head(1,2), and found that stun lookups were failing with a "Timeout" error. Debugging the issue points to memory alignment issues specific to the ARM architecture. Basically, due to the fact that the uint8_t array "buf", is not aligned at a 32-bit memory boundary, the stun lookup packet sent to the stun server was malformed. Web searches confirm that a symptom of misaligned access on ARM is an apparent memory corruption. This can be fixed by forcing alignment of specific variables to 32-bit boundary. As an example, in the switch_stun_lookup function of file src/switch_stun.c , we currently have { switch_sockaddr_t *local_addr = NULL, *remote_addr = NULL, *from_addr = NULL; switch_socket_t *sock = NULL; uint8_t buf[260] = { 0 }; .... } If I change it to { switch_sockaddr_t *local_addr = NULL, *remote_addr = NULL, *from_addr = NULL; switch_socket_t *sock = NULL; uint8_t buf[260] __attribute__ (aligned(4)) = { 0 }; .... } or { switch_sockaddr_t *local_addr = NULL, *remote_addr = NULL, *from_addr = NULL; switch_socket_t *sock = NULL; uint8_t *buf; buf = (uint8_t*) malloc(sizeof(uint8_t)*260); .... } the stun lookup packet sent is correct.freeswitch is also able to parse the recvd packet and correctly determines the external IP. Random crashes coupled with looking at the code in switch_rtp suggests that the memory alignment issue may not be limited to this function or even this file. Has anyone looked at ARM memory alignment issues for freeswitch in general? cheers, -m (1) Current head leads to the situation where call gets connected but there is no-audio. I have not had the chance to debug this, but suspect this could be due to the "eat rtp .... " patch combined with ARM memory alignment issues. (2) The tested (audio working, some issues) git version is commit 715d250e171a94736b19019ac742f739899ad997 Author: Mathieu Parent <math.parent@gmail.com> Date: Wed Dec 15 21:29:52 2010 +0100
take a look at swith_types.h 645-710. This is what you would need to do cross platform to align stuff. I am surprised the first way isn't aligned already, the first proposed change wont be cross platform, the second, if that works, should be fine (as long as you free it later of course) but I am unsure if that will actually work cross platform. The trick here is to figure out how to fix it once for all platforms. Mike On Jan 18, 2011, at 12:35 PM, M D wrote:
Summary: Has anyone looked at/actively looking at the 32-bit boundary memory alignment requirement for the ARM architecture?
Detail: I am running freeswitch on an ARM platform (dockstar based on Marvell Kirkwood). I have compiled and tested both freeswitch-1.0.6 and freeswitch-head(1,2), and found that stun lookups were failing with a "Timeout" error. Debugging the issue points to memory alignment issues specific to the ARM architecture. Basically, due to the fact that the uint8_t array "buf", is not aligned at a 32-bit memory boundary, the stun lookup packet sent to the stun server was malformed. Web searches confirm that a symptom of misaligned access on ARM is an apparent memory corruption.
This can be fixed by forcing alignment of specific variables to 32-bit boundary. As an example, in the switch_stun_lookup function of file src/switch_stun.c , we currently have
{ switch_sockaddr_t *local_addr = NULL, *remote_addr = NULL, *from_addr = NULL; switch_socket_t *sock = NULL; uint8_t buf[260] = { 0 }; .... }
If I change it to { switch_sockaddr_t *local_addr = NULL, *remote_addr = NULL, *from_addr = NULL; switch_socket_t *sock = NULL; uint8_t buf[260] __attribute__ (aligned(4)) = { 0 }; .... }
or { switch_sockaddr_t *local_addr = NULL, *remote_addr = NULL, *from_addr = NULL; switch_socket_t *sock = NULL; uint8_t *buf; buf = (uint8_t*) malloc(sizeof(uint8_t)*260); .... }
the stun lookup packet sent is correct.freeswitch is also able to parse the recvd packet and correctly determines the external IP.
Random crashes coupled with looking at the code in switch_rtp suggests that the memory alignment issue may not be limited to this function or even this file. Has anyone looked at ARM memory alignment issues for freeswitch in general?
cheers, -m
(1) Current head leads to the situation where call gets connected but there is no-audio. I have not had the chance to debug this, but suspect this could be due to the "eat rtp .... " patch combined with ARM memory alignment issues.
(2) The tested (audio working, some issues) git version is commit 715d250e171a94736b19019ac742f739899ad997 Author: Mathieu Parent <math.parent@gmail.com> Date: Wed Dec 15 21:29:52 2010 +0100
_______________________________________________ FreeSWITCH-dev mailing list FreeSWITCH-dev@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-dev http://www.freeswitch.org
On Tue, 18 Jan 2011 12:54:53 -0500 Michael Jerris <mike@jerris.com> wrote: In general, should I conclude that this issue has not been encountered before?
take a look at swith_types.h 645-710. This is what you would need to do cross platform to align stuff. If I understood you correctly, you mean to point out a general cross platform issue and how it is dealt with withing freeswitch code. Right? Just to complete the info, the ARM processor is running Little-Endian. The code/settings/dialplan worked fine on a x86 linux machine, also little endian. Probably nothing new, but I thought I should fill in the details.
I am surprised the first way isn't aligned already, the first proposed change wont be cross platform, the second, if that works, should be fine (as long as you free it later of course) but I am unsure if that will actually work cross platform. The trick here is to figure out how to fix it once for all platforms.
Mike
I expected that the original code would be aligned too. It just wasn't, atleast with gcc 4.3.3 + Code Sourcery enhancements. In gcc 4.4.5 ( I think, I don't have access to my development machine at the moment.) using the switch -falign-functions did align the buffer buf at a 32-bit boundary. Of course align-functions does not ensure that all uint8_t buffers would be aligned. Perhaps a conditional define would be more suitable fix, considering all platforms. As far as I can tell, the memory alignment requirement is ARM specific, which restricts this issue to gcc/linux. So in pseudo code #define ALIGN_ON_ARM ifdef GCC ifdef arm #undef ALIGN_ON_ARM #define ALIGN_ON_ARM __attribute__ (aligned(4)) #endif #endif and then in the source code uint8_t buf[260] ALIGN_ON_ARM = {0}; Would something along these lines work? The only issue is that it would touch a lot of the code. The use of structures with a uint8 buffer is likely widely used within freeswitch. -m
On Jan 18, 2011, at 12:35 PM, M D wrote:
Summary: Has anyone looked at/actively looking at the 32-bit boundary memory alignment requirement for the ARM architecture?
On Jan 18, 2011, at 2:42 PM, M D wrote:
On Tue, 18 Jan 2011 12:54:53 -0500 Michael Jerris <mike@jerris.com> wrote:
In general, should I conclude that this issue has not been encountered before?
This has certainly been encountered before, on different platforms, including arm. I suspect this is a toolchain issue more than anything in this case as we have worked on arm for quite some time.
take a look at swith_types.h 645-710. This is what you would need to do cross platform to align stuff. If I understood you correctly, you mean to point out a general cross platform issue and how it is dealt with withing freeswitch code. Right? Just to complete the info, the ARM processor is running Little-Endian. The code/settings/dialplan worked fine on a x86 linux machine, also little endian. Probably nothing new, but I thought I should fill in the details.
the trick is to fix it in the way that fixes it for the most things all at once.
I am surprised the first way isn't aligned already, the first proposed change wont be cross platform, the second, if that works, should be fine (as long as you free it later of course) but I am unsure if that will actually work cross platform. The trick here is to figure out how to fix it once for all platforms.
Mike
I expected that the original code would be aligned too. It just wasn't, atleast with gcc 4.3.3 + Code Sourcery enhancements. In gcc 4.4.5 ( I think, I don't have access to my development machine at the moment.) using the switch -falign-functions did align the buffer buf at a 32-bit boundary. Of course align-functions does not ensure that all uint8_t buffers would be aligned.
Is there a gcc arg that will make this work? If so we can test for that being available and use it when it is.
Perhaps a conditional define would be more suitable fix, considering all platforms. As far as I can tell, the memory alignment requirement is ARM specific, which restricts this issue to gcc/linux. So in pseudo code
#define ALIGN_ON_ARM ifdef GCC ifdef arm #undef ALIGN_ON_ARM #define ALIGN_ON_ARM __attribute__ (aligned(4)) #endif #endif
and then in the source code
uint8_t buf[260] ALIGN_ON_ARM = {0};
Would something along these lines work? The only issue is that it would touch a lot of the code. The use of structures with a uint8 buffer is likely widely used within freeswitch.
I would much rather not litter the code with a ton of garbage like this, there should be a way to tell the compiler how we want alignment to work either with a single ifdef in a header, or with a compiler arg.
-m
On Jan 18, 2011, at 12:35 PM, M D wrote:
Summary: Has anyone looked at/actively looking at the 32-bit boundary memory alignment requirement for the ARM architecture?
On Tue, 18 Jan 2011 15:40:40 -0500 Michael Jerris <mike@jerris.com> wrote:
On Jan 18, 2011, at 2:42 PM, M D wrote:
On Tue, 18 Jan 2011 12:54:53 -0500 Michael Jerris <mike@jerris.com> wrote:
In general, should I conclude that this issue has not been encountered before?
This has certainly been encountered before, on different platforms, including arm. I suspect this is a toolchain issue more than anything in this case as we have worked on arm for quite some time.
Would it be possible to know which tuple works? I.e. binutils,gcc, libc (glibc or uclibc). At the moment I am trying to build different combinations of the toolchain to see if I can isolate a working version. What is the best way to share a testcase with the list?
I expected that the original code would be aligned too. It just wasn't, atleast with gcc 4.3.3 + Code Sourcery enhancements. In gcc 4.4.5 ( I think, I don't have access to my development machine at the moment.) using the switch -falign-functions did align the buffer buf at a 32-bit boundary. Of course align-functions does not ensure that all uint8_t buffers would be aligned.
Is there a gcc arg that will make this work? If so we can test for that being available and use it when it is. I could not find one. I think I understand why gcc would not align the uint8_t array to a 16/32-bit boundary: The alignment is data type dependent, and since uint8_t is 8-bits, there is no need to align it with a 32-bit boundary.
For what it is worth, malloc documentation suggests that it will return memory suitably aligned for **any** variable type, which means that it should work correctly on all platforms. I am not sure if the run-time overhead would be prohibitively high. Thanks for your input on all this. -m
On Tue, 18 Jan 2011 15:40:40 -0500 Michael Jerris <mike@jerris.com> wrote:
On Jan 18, 2011, at 2:42 PM, M D wrote:
On Tue, 18 Jan 2011 12:54:53 -0500 Michael Jerris <mike@jerris.com> wrote:
In general, should I conclude that this issue has not been encountered before?
This has certainly been encountered before, on different platforms, including arm. I suspect this is a toolchain issue more than anything in this case as we have worked on arm for quite some time.
Sorry for the noise. Looks like a gcc-4.3 issue. I have tested the following code with gcc-4.3.3, gcc-4.3.5 and gcc-4.4.4 and gcc-4.4.5. gcc-4.4.4 comes from gentoo crossdev environment, everything else is from openwrt build system. The 4.4 versions align the buf2 in the code below with the 32-bit boundary while the 4.3 versions don't. All test cases compiled with -march=armv5te. No other compiler flags/switches used. Optimization options -Os vs -O2 do not seem to make a difference. If anyone has ready access to arm-gcc and an arm machine, I would be curious to see my experience is consistent with others'. cheers, -m P.S Code from memory. ----- test_align.c ---- #include<stdio.h> #include<stdlib.h> #include <string.h> #include<netinet/in.h> /* typedef struct { uint16_t type; uint16_t length; char id[16]; } switch_stun_packet_header_t; */ void main(void) { uint8_t buf1[260] = {0}; char j; uint8_t buf2[260] = {0}; //switch_stun_packet_header_t *header; printf("buf1 starts at %x\n",buf1); printf("j starts at %x\n",&j); printf("buf2 starts at %x\n",buf2); } ------------------------------
Dear Michael, On Tue, Jan 18, 2011 at 3:40 PM, Michael Jerris <mike@jerris.com> wrote:
On Jan 18, 2011, at 2:42 PM, M D wrote:
On Tue, 18 Jan 2011 12:54:53 -0500 Michael Jerris <mike@jerris.com> wrote:
In general, should I conclude that this issue has not been encountered before?
This has certainly been encountered before, on different platforms, including arm. I suspect this is a toolchain issue more than anything in this case as we have worked on arm for quite some time.
I have done some further testing and it appears that the issue is not necessarily a toolchain issue. It is true that some toolchains seem to exacerbate the issue, but even with gcc-4.4.5 some memory alignment problems seem to be still there. The linux kernel has an alignment trap on ARM cpus. It is possible to activate alignment fix-ups at the kernel level by executing echo 2 > /proc/cpu/alignment Restarting freeswitch after the alignment trap is active shows cat /proc/cpu/alignment User: 25068 System: 1 Skipped: 0 Half: 28 Word: 0 DWord: 25041 Multi: 0 User faults: 2 (fixup) My tests indicate that majority of the alignment traps are triggered from freeswitch code. I would really appreciate if anyone can provide any information on successful builds on ARM. Information regarding the build system used would be invaluable in isolating the cause. The page at http://lecs.cs.ucla.edu/wiki/index.php/XScale_alignment has some general details regarding memory alignment on ARM. -m
participants (2)
-
M D
-
Michael Jerris