Openblas: Compile for iOS

Created on 13 May 2015  路  41Comments  路  Source: xianyi/OpenBLAS

I need to compile this library for iPhone. Target architecture is arm64 and armv7. I set default clang compiler to use and arm64 arch.

make CC=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang ARCH=arm64

It fails with error:
make[1]: *** No rule to make target "../kernel/arm64/amax.S", needed by "samax_k.o". Stop.

It's successfully compile if I do not set ARCH=arm64. Can you give advice how to compile library with arm64 and armv7 support.

Also if I add arch flag to compiler, so compile going like so:

Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang -O2 -arch arm64 -miphoneos-version-min=5.0 -DMAX_STACK_ALLOC=2048 -DEXPRECISION  -Wall -m64 -DF_INTERFACE_GFORT -fPIC -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=8 -DASMNAME=_saxpy -DASMFNAME=_saxpy_ -DNAME=saxpy_ -DCNAME=saxpy -DCHAR_NAME=\"saxpy_\" -DCHAR_CNAME=\"saxpy\" -DNO_AFFINITY -I.. -I. -UDOUBLE  -UCOMPLEX -c axpy.c -o saxpy.o

I start get error like this.

/usr/include/sys/cdefs.h:680:2: error: Unsupported architecture
/usr/include/sys/cdefs.h:680:2#error Unsupported architecture:
/usr/include/sys/cdefs.h:680:2: error: Unsupported architecture
/usr/include/sys/cdefs.h:680:2: error: Unsupported architecture
unknown type name '__uint32_t'; did you mean '__uint128_t'?
error: unknown type name '__darwin_va_list'
error: unknown type name '__darwin_size_t'
// and others

Thank you.

picture AlexandrGraschenkov

All 41 comments

Closed ticket #523 looks relevant. Also please try the develop branch if you are currently using one of the releases - I believe some ARM-related fixes went in after 0.2.14

martin-frbg picture martin-frbg on 13 May 2015

@AlexandrGraschenkov , I didn't try to compile OpenBLAS for iOS before.

xianyi picture xianyi on 13 May 2015

Thank you for reply. I've try to make TARGET=ARMV8 but get error on compile:

symm.c:388:36: error: use of undeclared identifier 'QGEMM_DEFAULT_P'
gemm.c:397:37: error: use of undeclared identifier 'QGEMM_DEFAULT_P'
gemm.c:397:46: error: use of undeclared identifier 'QGEMM_DEFAULT_Q'
// and so on

With make TARGET=ARMV8 ARCH=arm64 more code get compiled but still get errors:

../kernel/arm64/../arm/scal.c:55:2: error: non-void function 'sscal_k' should return a value [-Wreturn-type]

After change to return 0; get errors:

../kernel/arm64/sgemm_kernel_4x4.S:759:19: error: unknown token in expression
        add sp,sp,#-(5*16)
                  ^
../kernel/arm64/sgemm_kernel_4x4.S:760:19: error: brackets expression not supported on this target
        stp d8,d9,[sp,#(0*16)]
                  ^
../kernel/arm64/sgemm_kernel_4x4.S:761:21: error: brackets expression not supported on this target
        stp d10,d11,[sp,#(1*16)]
                    ^
../kernel/arm64/sgemm_kernel_4x4.S:762:21: error: brackets expression not supported on this target
        stp d12,d13,[sp,#(2*16)]
                    ^
../kernel/arm64/sgemm_kernel_4x4.S:763:21: error: brackets expression not supported on this target
        stp d14,d15,[sp,#(3*16)]
                    ^
../kernel/arm64/sgemm_kernel_4x4.S:764:21: error: brackets expression not supported on this target
        stp d16,d17,[sp,#(4*16)]
                    ^
../kernel/arm64/sgemm_kernel_4x4.S:766:22: error: unexpected token in argument list
        mov x12, v0.d[0]
// and so on
AlexandrGraschenkov picture AlexandrGraschenkov on 13 May 2015

Googling for the "brackets expression..." error message finds some cases where the cause appeared to be mis-detection of the target architecture, e.g.
https://github.com/project-imas/security-check/issues/3
(I assume you did issue "make clean" between your various attempts ?)

martin-frbg picture martin-frbg on 13 May 2015

Hi Alexander,

Have you solved the compilation problem? I have exactly the same issue.

tchernitski picture tchernitski on 15 Jun 2015

Hi @tchernitski
I'm not solved the problem. I compile one project witch uses Open-BLAS. I decide to complie project without library, and change code usage of Open-BLAS to Accelerate framework functions. But project successfully compile for iOS, and I stop dig into this problem.

AlexandrGraschenkov picture AlexandrGraschenkov on 15 Jun 2015

Thanks for answer, but I would like to use something that is less platform-dependend

tchernitski picture tchernitski on 16 Jun 2015

Hello,

Do you have any progress of this issue?

sstepashka picture sstepashka on 1 Apr 2016

Hi,
I replaced OpenBlas by Eigen in my project

tchernitski picture tchernitski on 1 Apr 2016

Hello,
Not sure if this still actual but here is how I did build for ARM64 arch.

  1. In file c_check add following line for $cross_suffix.
    $cross_suffix = "";

    if (dirname($compiler_name) ne ".") {
        $cross_suffix .= dirname($compiler_name) . "/";
    }

    if (basename($compiler_name) =~ /([^\s]*-)(.*)/) {
        $cross_suffix .= $1;
    }

    $cross_suffix = "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/";

You need this because we will provide CC with arguments and Perl function for extracting directory doesn't work well.

  1. Create file with following content.

    TOOLCHAIN_PATH=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
SYSROOT_PATH=/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS10.2.sdk
make TARGET=ARMV8 BINARY=64 HOSTCC=clang CC="$TOOLCHAIN_PATH/clang -isysroot $SYSROOT_PATH -arch arm64" NOFORTRAN=1 libs

Now you can build library.

ivan-ushakov picture ivan-ushakov on 26 Feb 2017

Thank you very much for this. I wonder if your addition to the c_check file can be made conditional on something like
if ( $data =~ /OS_DARWIN/ && $data =~ /ARCH_ARM/)
which should allow adding it to the current code without breaking other platforms ? Then users would only need to create your build script (which I think could be added as "quickbuild.ios", if the toolchain path can be expected to be fairly constant). Unfortunately I do not have any Apple hardware to try this myself.

martin-frbg picture martin-frbg on 26 Feb 2017

I guess we need to find how get directory name from path like this: 

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS10.2.sdk -arch arm64

Functions like dirname() or basename() don't work well for such path.

ivan-ushakov picture ivan-ushakov on 28 Feb 2017

I understand the sysroot path will contain a version number and/or target platform name, so your build script would always need to be edited by the user. Can we rely on the toolchain path that you put in c_check to be constant ? Then at least the user would not have to change any files from OpenBLAS if that line was present by default - less chance to get something wrong.

martin-frbg picture martin-frbg on 28 Feb 2017

Copied your text to the wiki as https://github.com/xianyi/OpenBLAS/wiki/How-to-build-OpenBLAS-for-iPhone-iOS

martin-frbg picture martin-frbg on 1 Mar 2017

Will your solution work for armv7?

sstepashka picture sstepashka on 1 Mar 2017

@sstepashka
No, I tried to build for armv7 arch but got error about CPU/FPU features.

ivan-ushakov picture ivan-ushakov on 1 Mar 2017

hi, all, I build the develop branches to use the solution, it has no error, but just generate .a library(the size only 3m) 锛寃hen I use "make" to build the x86 library锛宨t generate 2 library(.a and .dylib, the size both more than 10m), I want to use the dylib
by the way, it has a warning:"clang: warning: using sysroot for 'iPhoneOS' but targeting 'MacOSX'"
is it right?

tamworth picture tamworth on 5 Mar 2017

@tamworth Did you build for armv7?

ccy022364 picture ccy022364 on 7 Mar 2017

I've found Accelerate.framework with BLAS functions. Maybe you shouldn't use OpenBLAS for iOS?
See https://developer.apple.com/reference/accelerate/blas

sstepashka picture sstepashka on 7 Mar 2017

Thanks,I kown Accelerate.framework.Now we want to test the Openblas and Accelerate.framework,which one is faster.

ccy022364 picture ccy022364 on 7 Mar 2017

@ccy022364 I build for arm64

tamworth picture tamworth on 7 Mar 2017

@tamworth googling for the error message led me to https://curl.haxx.se/mail/lib-2016-10/0094.html which seems to suggest you may have "-mmacosx-version-min" or similar set somewhere in your default clang options (but I know nothing about cross-compiling for iOS on OSX)
And if someone of you who tried to build this for armv7 could post the error message (about wrong or missing CPU features apparently ?) maybe this will provide a clue for the ARM experts here

martin-frbg picture martin-frbg on 7 Mar 2017

When I use arm64 library,it failed with error,
Undefined symbols for architecture arm64:
"_sgemm_kernel", referenced from:
_sgemm_nn in libopenblas_armv8p-r0.2.20.dev.a(sgemm_nn.o)
_sgemm_nt in libopenblas_armv8p-r0.2.20.dev.a(sgemm_nt.o)
_inner_thread in libopenblas_armv8p-r0.2.20.dev.a(sgemm_thread_nn.o)
_inner_thread in libopenblas_armv8p-r0.2.20.dev.a(sgemm_thread_nt.o)
_inner_thread in libopenblas_armv8p-r0.2.20.dev.a(sgemm_thread_tn.o)
_inner_thread in libopenblas_armv8p-r0.2.20.dev.a(sgemm_thread_tt.o)
_sgemm_tn in libopenblas_armv8p-r0.2.20.dev.a(sgemm_tn.o)
...
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

thank you

ccy022364 picture ccy022364 on 8 Mar 2017

@martin-frbg Thank you for your reply, it works, this is my script:
TOOLCHAIN_PATH=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin SYSROOT_PATH=/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS10.2.sdk make clean make TARGET=ARMV8 BINARY=64 HOSTCC=clang CC="$TOOLCHAIN_PATH/clang -isysroot $SYSROOT_PATH -arch arm64 -miphoneos-version-min=8.0 -O2" NOFORTRAN=1 libs

tamworth picture tamworth on 10 Mar 2017

@ccy022364 This is my test project, It working on arm64 device https://github.com/tamworth/BLASCompare

In this project, I use same function to make a test between openblas and Accelerate.framewrok, It seems that openblas is far slower than Accelerate.framewrok, I'm a newbie of openblas, I don't know the results whether it's right

tamworth picture tamworth on 11 Mar 2017
👍1

Try building OpenBLAS for TARGET=CORTEXA57 and compare the the performance on arm64.

Also what APIs are you comparing??

ashwinyes picture ashwinyes on 11 Mar 2017

@ashwinyes I comparing with Accelerate.framewrok, it's Apple's cblas library,
I run 10000000 times on iPhone 5S with: 

    double A[6] = {1.0,2.0,1.0,-3.0,4.0,-1.0};
    double B[6] = {1.0,2.0,1.0,-3.0,4.0,-1.0};
    double C[9] = {.5,.5,.5,.5,.5,.5,.5,.5,.5};
    cblas_dgemm(CblasColMajor, CblasNoTrans, CblasTrans,3,3,2,1,A, 3, B, 3,2,C,3);

It took 9s on openblas, and it only took 2s on Accelerate.framewrok

I have try to build withTARGET=CORTEXA57, It caused so many error, And I'm not sure the CORTEXA57 is applicable to iPhone

tamworth picture tamworth on 11 Mar 2017

Thanks for testing.

CORTEXA57 should work on all arm64 processors having Advanced SIMD support which I believe should be there for iphone.

Could you share the compile errors that you are seeing for CORTEXA57 target ??

FYI. TARGET=ARM64 just uses the generic C implementations. So not much performance is to be expected. On actual cortexa57 processor, TARGET=CORTEXA57 is 3-4x times faster than TARGET=ARM64 if i remember correctly.

ashwinyes picture ashwinyes on 11 Mar 2017

Also I believe the matrix sizes that you are using is very small and not appropriate for benchmarking OpenBLAS as it is optimized for larger matrices.

Would request you to try with larger matrix sizes (starting from 100x100) with lesser number if iterations.

ashwinyes picture ashwinyes on 11 Mar 2017

@ashwinyes Here is the error message

<instantiation>:4:21: error: unexpected token in argument list
 beq KERNEL_F1_NEXT_\@
                    ^
<instantiation>:1:1: note: while in macro instantiation
KERNEL_F1
^
../kernel/arm64/nrm2.S:150:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:7:27: error: unexpected token in argument list
 bge KERNEL_F1_SCALE_GE_X_\@
                          ^
<instantiation>:1:1: note: while in macro instantiation
KERNEL_F1
^
../kernel/arm64/nrm2.S:150:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:13:19: error: unexpected token in argument list
 b KERNEL_F1_NEXT_\@
                  ^
<instantiation>:1:1: note: while in macro instantiation
KERNEL_F1
^
../kernel/arm64/nrm2.S:150:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:14:22: error: unknown token in expression
KERNEL_F1_SCALE_GE_X_\@:
                     ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS10.2.sdk -arch arm64 -miphoneos-version-min=8.0 -O2 -c -O2 -DMAX_STACK_ALLOC=2048 -Wall -DF_INTERFACE_GFORT -fPIC -DNO_LAPACK -DNO_LAPACKE -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=4 -march=armv8-a+crc+crypto+fp+simd -mtune=cortex-a57 -DASMNAME=_saxpby_k -DASMFNAME=_saxpby_k_ -DNAME=saxpby_k_ -DCNAME=saxpby_k -DCHAR_NAME=\"saxpby_k_\" -DCHAR_CNAME=\"saxpby_k\" -DNO_AFFINITY -I.. -UDOUBLE  -UCOMPLEX -UCOMPLEX -UDOUBLE ../kernel/arm64/../arm/axpby.c -o saxpby_k.o
<instantiation>:1:1: note: while in macro instantiation
KERNEL_F1
^
../kernel/arm64/nrm2.S:150:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:14:22: error: invalid operand
KERNEL_F1_SCALE_GE_X_\@:
                     ^
<instantiation>:1:1: note: while in macro instantiation
KERNEL_F1
^
../kernel/arm64/nrm2.S:150:2: note: while in macro instantiation
 KERNEL_F8
 ^
../kernel/arm64/nrm2.S:87:16: error: unknown token in expression
KERNEL_F1_NEXT_\@:
               ^
<instantiation>:1:1: note: while in macro instantiation
KERNEL_F1
^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
../kernel/arm64/nrm2.S:87:16: error: invalid operand
KERNEL_F1_NEXT_\@:
               ^
<instantiation>:1:1: note: while in macro instantiation
KERNEL_F1
^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:4:21: error: unexpected token in argument list
 beq KERNEL_F1_NEXT_\@
                    ^
<instantiation>:2:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:7:27: error: unexpected token in argument list
 bge KERNEL_F1_SCALE_GE_X_\@
                          ^
<instantiation>:2:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:13:19: error: unexpected token in argument list
 b KERNEL_F1_NEXT_\@
                  ^
<instantiation>:2:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:14:22: error: unknown token in expression
KERNEL_F1_SCALE_GE_X_\@:
                     ^
<instantiation>:2:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:14:22: error: invalid operand
KERNEL_F1_SCALE_GE_X_\@:
                     ^
<instantiation>:2:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
../kernel/arm64/nrm2.S:87:16: error: unknown token in expression
KERNEL_F1_NEXT_\@:
               ^
<instantiation>:2:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
../kernel/arm64/nrm2.S:87:16: error: invalid operand
KERNEL_F1_NEXT_\@:
               ^
<instantiation>:2:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:4:21: error: unexpected token in argument list
 beq KERNEL_F1_NEXT_\@
                    ^
<instantiation>:3:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:7:27: error: unexpected token in argument list
 bge KERNEL_F1_SCALE_GE_X_\@
                          ^
<instantiation>:3:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:13:19: error: unexpected token in argument list
 b KERNEL_F1_NEXT_\@
                  ^
<instantiation>:3:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:14:22: error: unknown token in expression
KERNEL_F1_SCALE_GE_X_\@:
                     ^
<instantiation>:3:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:14:22: error: invalid operand
KERNEL_F1_SCALE_GE_X_\@:
                     ^
<instantiation>:3:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
../kernel/arm64/nrm2.S:87:16: error: unknown token in expression
KERNEL_F1_NEXT_\@:
               ^
<instantiation>:3:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
../kernel/arm64/nrm2.S:87:16: error: invalid operand
KERNEL_F1_NEXT_\@:
               ^
<instantiation>:3:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:4:21: error: unexpected token in argument list
 beq KERNEL_F1_NEXT_\@
                    ^
<instantiation>:4:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:7:27: error: unexpected token in argument list
 bge KERNEL_F1_SCALE_GE_X_\@
                          ^
<instantiation>:4:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:13:19: error: unexpected token in argument list
 b KERNEL_F1_NEXT_\@
                  ^
<instantiation>:4:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:14:22: error: unknown token in expression
KERNEL_F1_SCALE_GE_X_\@:
                     ^
<instantiation>:4:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:14:22: error: invalid operand
KERNEL_F1_SCALE_GE_X_\@:
                     ^
<instantiation>:4:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
../kernel/arm64/nrm2.S:87:16: error: unknown token in expression
KERNEL_F1_NEXT_\@:
               ^
<instantiation>:4:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
../kernel/arm64/nrm2.S:87:16: error: invalid operand
KERNEL_F1_NEXT_\@:
               ^
<instantiation>:4:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:4:21: error: unexpected token in argument list
 beq KERNEL_F1_NEXT_\@
                    ^
<instantiation>:5:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:7:27: error: unexpected token in argument list
 bge KERNEL_F1_SCALE_GE_X_\@
                          ^
<instantiation>:5:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:13:19: error: unexpected token in argument list
 b KERNEL_F1_NEXT_\@
                  ^
<instantiation>:5:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:14:22: error: unknown token in expression
KERNEL_F1_SCALE_GE_X_\@:
                     ^
<instantiation>:5:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:14:22: error: invalid operand
KERNEL_F1_SCALE_GE_X_\@:
                     ^
<instantiation>:5:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
../kernel/arm64/nrm2.S:87:16: error: unknown token in expression
KERNEL_F1_NEXT_\@:
               ^
<instantiation>:5:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
../kernel/arm64/nrm2.S:87:16: error: invalid operand
KERNEL_F1_NEXT_\@:
               ^
<instantiation>:5:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:4:21: error: unexpected token in argument list
 beq KERNEL_F1_NEXT_\@
                    ^
<instantiation>:6:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:7:27: error: unexpected token in argument list
 bge KERNEL_F1_SCALE_GE_X_\@
                          ^
<instantiation>:6:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:13:19: error: unexpected token in argument list
 b KERNEL_F1_NEXT_\@
                  ^
<instantiation>:6:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:14:22: error: unknown token in expression
KERNEL_F1_SCALE_GE_X_\@:
                     ^
<instantiation>:6:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:14:22: error: invalid operand
KERNEL_F1_SCALE_GE_X_\@:
                     ^
<instantiation>:6:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
../kernel/arm64/nrm2.S:87:16: error: unknown token in expression
KERNEL_F1_NEXT_\@:
               ^
<instantiation>:6:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
../kernel/arm64/nrm2.S:87:16: error: invalid operand
KERNEL_F1_NEXT_\@:
               ^
<instantiation>:6:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:4:21: error: unexpected token in argument list
 beq KERNEL_F1_NEXT_\@
                    ^
<instantiation>:7:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:7:27: error: unexpected token in argument list
 bge KERNEL_F1_SCALE_GE_X_\@
                          ^
<instantiation>:7:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:13:19: error: unexpected token in argument list
 b KERNEL_F1_NEXT_\@
                  ^
<instantiation>:7:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:14:22: error: unknown token in expression
KERNEL_F1_SCALE_GE_X_\@:
                     ^
<instantiation>:7:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:14:22: error: invalid operand
KERNEL_F1_SCALE_GE_X_\@:
                     ^
<instantiation>:7:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^                            pplications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS10.2.sdk -arch arm64 -miphoneos-version-min=8.0 -O2 -c -O2 -DMAX_STACK_ALLOC=2048 -Wall -DF_INTERFACE_GFORT -fPIC -DNO_LAPACK -DNO_LAPACKE -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=4 -march=armv8-a+crc+crypto+fp+simd -mtune=cortex-a57 -DASMNAME=_sgemv_n -DASMFNAME=_sgemv_n_ -DNAME=sgemv_n_ -DCNAME=sgemv_n -DCHAR_NAME=\"sgemv_n_\" -DCHAR_CNAME=\"sgemv_n\" -DNO_AFFINITY -I.. -UDOUBLE  -UCOMPLEX -UDOUBLE -UCOMPLEX  -UTRANS ../kernel/arm64/gemv_n.S -o sgemv_n.o
31merror: unknown token in expression
KERNEL_F1_NEXT_\@:
               ^
<instantiation>:7:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
../kernel/arm64/nrm2.S:87:16: error: invalid operand
KERNEL_F1_NEXT_\@:
               ^
<instantiation>:7:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:4:21: error: unexpected token in argument list
 beq KERNEL_F1_NEXT_\@
                    ^
<instantiation>:8:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:7:27: error: unexpected token in argument list
 bge KERNEL_F1_SCALE_GE_X_\@
                          ^
<instantiation>:8:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:13:19: error: unexpected token in argument list
 b KERNEL_F1_NEXT_\@
                  ^
<instantiation>:8:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:14:22: error: unknown token in expression
KERNEL_F1_SCALE_GE_X_\@:
                     ^
<instantiation>:8:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:14:22: error: invalid operand
KERNEL_F1_SCALE_GE_X_\@:
                     ^
<instantiation>:8:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
../kernel/arm64/nrm2.S:87:16: error: unknown token in expression
KERNEL_F1_NEXT_\@:
               ^
<instantiation>:8:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
../kernel/arm64/nrm2.S:87:16: error: invalid operand
KERNEL_F1_NEXT_\@:
               ^
<instantiation>:8:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:4:21: error: unexpected token in argument list
 beq KERNEL_F1_NEXT_\@
                    ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:159:2: note: while in macro instantiation
 KERNEL_F1
 ^
<instantiation>:7:27: error: unexpected token in argument list
 bge KERNEL_F1_SCALE_GE_X_\@
                          ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:159:2: note: while in macro instantiation
 KERNEL_F1
 ^
<instantiation>:13:19: error: unexpected token in argument list
 b KERNEL_F1_NEXT_\@
                  ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:159:2: note: while in macro instantiation
 KERNEL_F1
 ^
<instantiation>:14:22: error: unknown token in expression
KERNEL_F1_SCALE_GE_X_\@:
                     ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS10.2.sdk -arch arm64 -miphoneos-version-min=8.0 -O2 -c -O2 -DMAX_STACK_ALLOC=2048 -Wall -DF_INTERFACE_GFORT -fPIC -DNO_LAPACK -DNO_LAPACKE -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=4 -march=armv8-a+crc+crypto+fp+simd -mtune=cortex-a57 -DASMNAME=_sgemv_t -DASMFNAME=_sgemv_t_ -DNAME=sgemv_t_ -DCNAME=sgemv_t -DCHAR_NAME=\"sgemv_t_\" -DCHAR_CNAME=\"sgemv_t\" -DNO_AFFINITY -I.. -UDOUBLE  -UCOMPLEX -UDOUBLE -UCOMPLEX  -DTRANS  ../kernel/arm64/gemv_t.S -o sgemv_t.o
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:159:2: note: while in macro instantiation
 KERNEL_F1
 ^
<instantiation>:14:22: error: invalid operand
KERNEL_F1_SCALE_GE_X_\@:
                     ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:159:2: note: while in macro instantiation
 KERNEL_F1
 ^
../kernel/arm64/nrm2.S:87:16: error: unknown token in expression
KERNEL_F1_NEXT_\@:
               ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:159:2: note: while in macro instantiation
 KERNEL_F1
 ^
../kernel/arm64/nrm2.S:87:16: error: invalid operand
KERNEL_F1_NEXT_\@:
               ^
/var/folders/d4/hc88wzv923s4b0ck9vpwrlm80000gp/T/nrm2-afe453.s:159:2: note: while in macro instantiation
 KERNEL_F1
 ^
make[1]: *** [snrm2_k.o] Error 1
make[1]: *** Waiting for unfinished jobs....
make: *** [libs] Error 1

I build script with:

TOOLCHAIN_PATH=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
SYSROOT_PATH=/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS10.2.sdk
make clean
make TARGET=CORTEXA57 BINARY=64 HOSTCC=clang CC="$TOOLCHAIN_PATH/clang -isysroot $SYSROOT_PATH -arch arm64 -miphoneos-version-min=8.0 -O2" NOFORTRAN=1  libs

Thank you

tamworth picture tamworth on 12 Mar 2017

Thanks for the logs.

Looks like the only files that are causing the errors are kernel/arm64/nrm2.S and kernel/arm64/znrm2.S. clang dont seem to understand the \@ token in macros used to create unique labels.

You can comment out the following lines in kernel/arm64/KERNEL.CORTEXA57 and do a clean make to get rid of the compilation errors.
SNRM2KERNEL = nrm2.S
DNRM2KERNEL = nrm2.S
CNRM2KERNEL = znrm2.S
ZNRM2KERNEL = znrm2.S

Thanks

ashwinyes picture ashwinyes on 12 Mar 2017
👍1

@ashwinyes Thank you for your reply, The library is successful compile with your solution, But when I run with Xcode, It caused some error:

Undefined symbols for architecture arm64:
  "_dgemm_incopy", referenced from:
      _inner_thread in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_thread_tn.o)
      _inner_thread in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_thread_tt.o)
      _dgemm_tn in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_tn.o)
      _dgemm_tt in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_tt.o)
  "_dgemm_otcopy", referenced from:
      _dgemm_nt in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_nt.o)
      _inner_thread in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_thread_nt.o)
      _inner_thread in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_thread_tt.o)
      _dgemm_tt in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_tt.o)
  "_dgemm_oncopy", referenced from:
      _dgemm_nn in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_nn.o)
      _inner_thread in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_thread_nn.o)
      _inner_thread in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_thread_tn.o)
      _dgemm_tn in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_tn.o)
  "_dgemm_itcopy", referenced from:
      _dgemm_nn in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_nn.o)
      _dgemm_nt in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_nt.o)
      _inner_thread in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_thread_nn.o)
      _inner_thread in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_thread_nt.o)
  "_dgemm_kernel", referenced from:
      _dgemm_nn in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_nn.o)
      _dgemm_nt in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_nt.o)
      _inner_thread in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_thread_nn.o)
      _inner_thread in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_thread_nt.o)
      _inner_thread in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_thread_tn.o)
      _inner_thread in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_thread_tt.o)
      _dgemm_tn in libopenblas_cortexa57p-r0.2.20.dev.a(dgemm_tn.o)
      ...
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
tamworth picture tamworth on 12 Mar 2017

Now this is a tougher problem.

All the above undefined references are implemented in assembly file for TARGET=CORTEXA57, but are implemented in C files for TARGET=ARMV8. So, I think the linker is unable to find any symbols implemented in assembly files.

I believe you will face the same issue if you try to use SGEMM in TARGET=ARMV8 as its implemented in assembly file.

Looks like OpenBLAS is lacking support for iOS compilation and this needs to be added. I don't have much idea how compilation/linking works in iOS. So will take time for me to figure out as I dont even have a development environment for iOS.

If any iOS experts are here, they will be able to figure this out quickly.

Thanks

ashwinyes picture ashwinyes on 12 Mar 2017
👍1

A quick fix that you can try is in the following branch.
https://github.com/ashwinyes/OpenBLAS/commits/ios_experiment_20170312
The above is based on http://stackoverflow.com/questions/28109826/arm64-using-gas-on-ios

Also, please note that the iOS ARM64 ABI has some divergences from the Standard Procedure Call Standard for the ARM 64-bit Architecture (AAPCS64) . Please refer below link.
https://developer.apple.com/library/content/documentation/Xcode/Conceptual/iPhoneOSABIReference/Articles/ARM64FunctionCallingConventions.html

This may result in unexpected errors as OpenBLAS assembly implementations for ARM64 does not follow the iOS ABI.

Thanks

ashwinyes picture ashwinyes on 12 Mar 2017
👍1

I wonder if this is only a problem with clang's built-in assembler, i.e. if adding "-no-integrated-as" to the CFLAGS would help in any way ?

martin-frbg picture martin-frbg on 12 Mar 2017

@martin-frbg No matter ARMV8 or CORTEXA57 also error caused When I used -no-integrated-as.
If I used CORTEXA57 and didn't comment out like this: https://github.com/ashwinyes/OpenBLAS/commit/db92f4ab791278a96c0a6ff2fe4c444f8f46dc80#diff-6b7ad97b93aa3fd3e259f94aafba00fb

the error is:

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/as: can't specifiy -Q with -arch arm64
clang: error: assembler command failed with exit code 1 (use -v to see invocation)
make[1]: *** [scopy.o] Error 1
make[1]: *** Waiting for unfinished jobs....
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/as: can't specifiy -Q with -arch arm64
clang: error: assembler command failed with exit code 1 (use -v to see invocation)
make[1]: *** [sscal.o] Error 1
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/as: can't specifiy -Q with -arch arm64
clang: error: assembler command failed with exit code 1 (use -v to see invocation)
make[1]: *** [sswap.o] Error 1
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/as: can't specifiy -Q with -arch arm64
clang: error: assembler command failed with exit code 1 (use -v to see invocation)
make[1]: *** [saxpy.o] Error 1
make: *** [libs] Error 1
tamworth picture tamworth on 13 Mar 2017

@ashwinyes I successfully build and run in ARMV8.
I used to print all functions with 'nm' in ARMV8 and CORTEXA57, Not any undefined symbols function in ARMV8, but it can be found in CORTEXA57 like this:

U _dgemm_incopy
U _dgemm_otcopy
tamworth picture tamworth on 13 Mar 2017

Please see the stackoverflow link that I posted in my previous comment.

iOS has an additional underscore at the beginning of the function name which is not followed in the arm64 assembly files. Please try the fix that I posted which adds this underscore to the function names in the assembly files.

ashwinyes picture ashwinyes on 13 Mar 2017

@tamworth Did you get the opportunity to test the code at https://github.com/ashwinyes/OpenBLAS/commits/ios_experiment_20170312

If yes, please let me know whether it worked for you.

Thanks

ashwinyes picture ashwinyes on 17 Mar 2017

@ashwinyes @tamworth Is it successful?

Thanks

brightbetter picture brightbetter on 16 Aug 2017

Hi, I am trying to make Openblas library to work on my project on IOS arm64 target:

@ccy022364 Did you manage to resolve your issue:

Showing All Messages
ld: symbol(s) not found for architecture arm64

"_sgemm_kernel", referenced from ...

I have following issue:
My project uses dlib with BLAS but linking fails with the same message.

Weird is that I used https://github.com/tamworth/BLASCompare project and Openblas library that I built behaves the same way as the one in that XCode project, it works without issues (also prebuilt Openblas library from that project fails to link in my project). Does anyone knows what might be the issue.

panic-milan picture panic-milan on 16 Nov 2017
Was this page helpful?
0 / 5 - 0 ratings

Related issues

yaswanthsvist picture yaswanthsvist  路  9Comments
conradsnicta picture conradsnicta  路  3Comments
aytekinar picture aytekinar  路  10Comments
cswluo picture cswluo  路  5Comments
nh2 picture nh2  路  9Comments