Quantcast

Problem with python bindings

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Problem with python bindings

Christian Boltz
Hello,

I'm one of the AppArmor developers, and we are using swig to provide
python bindings for libapparmor.

Unfortunately this broke since openSUSE Tumbleweed upgraded from swig
3.8.8 to 3.0.10:

===========================
root# aa-logprof
Traceback (most recent call last):
  File "/usr/lib64/python3.5/site-packages/LibAppArmor/__init__.py", line 18, in swig_import_helper
    return importlib.import_module(mname)
  File "/usr/lib64/python3.5/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 986, in _gcd_import
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 956, in _find_and_load_unlocked
ImportError: No module named '_LibAppArmor'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/sbin/aa-logprof", line 18, in <module>
    import apparmor.aa as apparmor
  File "/usr/lib/python3.5/site-packages/apparmor/aa.py", line 29, in <module>
    import apparmor.logparser
  File "/usr/lib/python3.5/site-packages/apparmor/logparser.py", line 19, in <module>
    import LibAppArmor
  File "/usr/lib64/python3.5/site-packages/LibAppArmor/__init__.py", line 21, in <module>
    _LibAppArmor = swig_import_helper()
  File "/usr/lib64/python3.5/site-packages/LibAppArmor/__init__.py", line 20, in swig_import_helper
    return importlib.import_module('_LibAppArmor')
  File "/usr/lib64/python3.5/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ImportError: No module named '_LibAppArmor'
===========================

You can find more details in
https://bugzilla.opensuse.org/show_bug.cgi?id=987607

and the code we use in
http://bazaar.launchpad.net/~apparmor-dev/apparmor/master/files/head:/libraries/libapparmor/swig/python/

Note that Makefile.am contains
    mv LibAppArmor.py __init__.py
which was added by another developer some years ago, so I don't know why
he did this. (Wild guess: to allow "import LibAppArmor" instead of
"import LibAppArmor.LibAppArmor", but that's just a guess)

It looks like the new way to find out the import path breaks by renaming
the LibAppArmor.py file :-(

Any ideas why this broke, and how to fix it?

BTW: My current workaround is to patch the generated LibAppArmor/__init__.py
to fix the import path, but this obviously isn't something I want as
official solution ;-)


Regards,

Christian Boltz
--
15:00:48 <bugbot> Meeting started [...] The chair is vuntz. [...]
15:01:58 <coolo> if vuntz is the chair, I'm the table :)
15:02:23 <vuntz> coolo: now tell me, is it best to be the table?
15:02:30 <vuntz> :-)
15:02:36 <Ilmehtar> can we put our feet up on the table?
[from #opensuse-project]


------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity planning
reports.http://sdm.link/zohodev2dev
_______________________________________________
Swig-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/swig-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Problem with python bindings

William S Fulton
Some background links for you regarding the python import changes in SWIG:

https://github.com/swig/swig/issues/691
https://github.com/swig/swig/pull/694
https://github.com/swig/swig/pull/702 - see the updated docs at
http://www.swig.org/Doc3.0/Python.html#Python_absrelimports. This is
probably going to be the most enlightening as your wrappers are
probably doing something unconventional wrt importing. Mike, could you
please advise Christian as to the best course of action as this is
your change?

Thanks
William

PS Fedora now runs all the projects that invoke SWIG during the build
when we make new SWIG releases. Can you get apparmor added as a Fedora
project and ensure it depends on SWIG, then it'll get tested whenever
we release.

On 26 July 2016 at 21:12, Christian Boltz <[hidden email]> wrote:

> Hello,
>
> I'm one of the AppArmor developers, and we are using swig to provide
> python bindings for libapparmor.
>
> Unfortunately this broke since openSUSE Tumbleweed upgraded from swig
> 3.8.8 to 3.0.10:
>
> ===========================
> root# aa-logprof
> Traceback (most recent call last):
>   File "/usr/lib64/python3.5/site-packages/LibAppArmor/__init__.py", line 18, in swig_import_helper
>     return importlib.import_module(mname)
>   File "/usr/lib64/python3.5/importlib/__init__.py", line 126, in import_module
>     return _bootstrap._gcd_import(name[level:], package, level)
>   File "<frozen importlib._bootstrap>", line 986, in _gcd_import
>   File "<frozen importlib._bootstrap>", line 969, in _find_and_load
>   File "<frozen importlib._bootstrap>", line 956, in _find_and_load_unlocked
> ImportError: No module named '_LibAppArmor'
>
> During handling of the above exception, another exception occurred:
>
> Traceback (most recent call last):
>   File "/usr/sbin/aa-logprof", line 18, in <module>
>     import apparmor.aa as apparmor
>   File "/usr/lib/python3.5/site-packages/apparmor/aa.py", line 29, in <module>
>     import apparmor.logparser
>   File "/usr/lib/python3.5/site-packages/apparmor/logparser.py", line 19, in <module>
>     import LibAppArmor
>   File "/usr/lib64/python3.5/site-packages/LibAppArmor/__init__.py", line 21, in <module>
>     _LibAppArmor = swig_import_helper()
>   File "/usr/lib64/python3.5/site-packages/LibAppArmor/__init__.py", line 20, in swig_import_helper
>     return importlib.import_module('_LibAppArmor')
>   File "/usr/lib64/python3.5/importlib/__init__.py", line 126, in import_module
>     return _bootstrap._gcd_import(name[level:], package, level)
> ImportError: No module named '_LibAppArmor'
> ===========================
>
> You can find more details in
> https://bugzilla.opensuse.org/show_bug.cgi?id=987607
>
> and the code we use in
> http://bazaar.launchpad.net/~apparmor-dev/apparmor/master/files/head:/libraries/libapparmor/swig/python/
>
> Note that Makefile.am contains
>     mv LibAppArmor.py __init__.py
> which was added by another developer some years ago, so I don't know why
> he did this. (Wild guess: to allow "import LibAppArmor" instead of
> "import LibAppArmor.LibAppArmor", but that's just a guess)
>
> It looks like the new way to find out the import path breaks by renaming
> the LibAppArmor.py file :-(
>
> Any ideas why this broke, and how to fix it?
>
> BTW: My current workaround is to patch the generated LibAppArmor/__init__.py
> to fix the import path, but this obviously isn't something I want as
> official solution ;-)
>
>
> Regards,
>
> Christian Boltz
> --
> 15:00:48 <bugbot> Meeting started [...] The chair is vuntz. [...]
> 15:01:58 <coolo> if vuntz is the chair, I'm the table :)
> 15:02:23 <vuntz> coolo: now tell me, is it best to be the table?
> 15:02:30 <vuntz> :-)
> 15:02:36 <Ilmehtar> can we put our feet up on the table?
> [from #opensuse-project]
>
>
> ------------------------------------------------------------------------------
> What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
> patterns at an interface-level. Reveals which users, apps, and protocols are
> consuming the most bandwidth. Provides multi-vendor support for NetFlow,
> J-Flow, sFlow and other flows. Make informed decisions using capacity planning
> reports.http://sdm.link/zohodev2dev
> _______________________________________________
> Swig-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/swig-user

------------------------------------------------------------------------------
_______________________________________________
Swig-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/swig-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Problem with python bindings

Christian Boltz
Hello,

thanks for the replies and the pointers. I'm quite new to swig, but I'm
always happy to learn something new ;-)

Am Freitag, 29. Juli 2016, 00:58:13 CEST schrieb [hidden email]:
> >>>>> " " == William S Fulton <[hidden email]> writes:

>     >> I'm one of the AppArmor developers, and we are using swig to
>     >> provide python bindings for libapparmor.

>     >> package, level) ImportError: No module named '_LibAppArmor'
>
>   Where exactly is this python module located?  This error is
> generated by python itself because the interpreter can not locate it
> either from the same package the swig shadow file was located in or
> as a global module.  My guess is that it is not being loaded as a
> shared object file (.so).  Or that _LibAppArmor.so is not located in
> either the global PYTHONPATH or the LibAppArmor package directory.

It really looks like it is "just" a location problem (see the workarounds
some paragraphs below).

With the swig-generated code and directory layout (also in the old
version), we would end up with having to use
    import LibAppArmor.LibAppArmor

I mentioned that Makefile.am renames LibAppArmor.py to __init__.py, which
means we can use
    import LibAppArmor
and this worked for some years (except with the latest swig version).

This probably also explains why things broke now (only) for AppArmor,
because swig doesn't expect that someone moves around its files.

I'd guess that the easiest solution would be a way that gives us a
location allowing to keep using
    import LibAppArmor
without having to rename or move files around.

>     >> ===========================
>     >>
>     >> You can find more details in
>     >> https://bugzilla.opensuse.org/show_bug.cgi?id=987607
>     >>
>     >> and the code we use in
>     >> http://bazaar.launchpad.net/~apparmor-dev/apparmor/master/files
>     >> /head:/libraries/libapparmor/swig/python/
>   Thanks.  I may have some time to look at the app armor source (after
> I finish up a couple of other projects).
>
>     >> Note that Makefile.am contains mv LibAppArmor.py __init__.py
>     >> which was added by another developer some years ago, so I don't
>     >> know why he did this. (Wild guess: to allow "import
>     >> LibAppArmor" instead of "import LibAppArmor.LibAppArmor", but
>     >> that's just a guess)
>
>   While this looks sorta dodgy, I don't think it is the reason python
> can't find _LibAppArmor in either the LibAppArmor package or the
> global namespace.  The problem is probably due to incorrect
> initialization of _LibAppArmor.

Please allow me to disagree ;-)  - a workaround is

   PYTHONPATH=/usr/lib64/python3.5/site-packages/LibAppArmor  aa-logprof

Another workaround is to patch the generated python module (which we
rename to __init__.py) to prepend "LibAppArmor." to mname:

    def swig_import_helper():
        import importlib
        pkg = __name__.rpartition('.')[0]
        mname = '.'.join((pkg, '_LibAppArmor')).lstrip('.')
+       mname = 'LibAppArmor.%s' % mname # <-- ADDED MANUALLY, HELPS :-)
        try:
            return importlib.import_module(mname)
        except ImportError:
            return importlib.import_module('_LibAppArmor')
    _LibAppArmor = swig_import_helper()
    del swig_import_helper


For comparison: older versions of swig generated code that only relied
on the directory name, but not on the filename:

from sys import version_info
if version_info >= (2,6,0):
    def swig_import_helper():
        from os.path import dirname
        import imp
        fp = None
        try:
            fp, pathname, description = imp.find_module('_LibAppArmor', [dirname(__file__)])  # <--- doesn't care if i's __init__.py or LibAppArmor.py
        except ImportError:
            import _LibAppArmor                                                                                                                                                                                                                                                
            return _LibAppArmor
        if fp is not None:
            try:
                _mod = imp.load_module('_LibAppArmor', fp, pathname, description)
            finally:
                fp.close()
            return _mod
    _LibAppArmor = swig_import_helper()
    del swig_import_helper


>   If my above guess about _LibAppArmor being statically linked is
> incorrect please point out the directory where _LibAppArmor.so is
> located in relative to LibAppArmor/__init__.py.

# rpm -ql python3-apparmor  |grep LibAppArmor
/usr/lib64/python3.5/site-packages/LibAppArmor
/usr/lib64/python3.5/site-packages/LibAppArmor-2.10.1-py3.5.egg-info
/usr/lib64/python3.5/site-packages/LibAppArmor/_LibAppArmor.cpython-35m-x86_64-linux-gnu.so
/usr/lib64/python3.5/site-packages/LibAppArmor/__init__.py
/usr/lib64/python3.5/site-packages/LibAppArmor/__pycache__
/usr/lib64/python3.5/site-packages/LibAppArmor/__pycache__/__init__.cpython-35.pyc

>     >> It looks like the new way to find out the import path breaks by
>     >> renaming the LibAppArmor.py file :-(
>
>   I don't think I would simply rename the file to __init__.py.  It
> seems better to have an __init__.py file that imports the symbols it
> wants to export in the package from LibAppArmor.py.  But I don't think
> this is the direct cause of the trouble.

See above, and creating an __init__.py with
    from LibAppArmor.LibAppArmor import *
might even be another workaround ;-)

>     >> Any ideas why this broke, and how to fix it?
>     >>
>     >> BTW: My current workaround is to patch the generated
>     >> LibAppArmor/__init__.py to fix the import path, but this
>     >> obviously isn't something I want as official solution ;-)
>
>   What change to PYTHONPATH causes python3 to locate _LibAppArmor.so?
> Or are you modifying the code in some other way?

See the workarounds above ;-)


Regards,

Christian Boltz
--
My "Irish" luck is fantastic, especially after a half-gallon
of Mexican Beer [Patrick Shanahan in opensuse-factory]


------------------------------------------------------------------------------
_______________________________________________
Swig-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/swig-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

How to cumulate %typemap(cscode) ?

RAVI NANJUNDAPPA
In reply to this post by William S Fulton
Hi,


I have two places in my .i files where I use %typemap(cscode) on the same class.
e.g.:
%typemap(cscode) MyClass %{
                // part #1...
%}

...

%typemap(cscode) MyClass %{
                // part #2...
%}

 
The problem is that they dont cumulate.  The last one wins !


I know the best would be to merge the two typemaps together but it’s impossible in my context.
(i.e.: the %typemap(cscode) is part of a macro that I call more than once on the same class !)

Is there a way to do it ?

Any help would be appreciated...

Thanks and Best Regards,
N Ravi
------------------------------------------------------------------------------

_______________________________________________
Swig-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/swig-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to cumulate %typemap(cscode) ?

RAVI NANJUNDAPPA
In reply to this post by William S Fulton
Any information on how to solve the below issue ?

Thanks and Best Regards,
N Ravi
 
--------- Original Message ---------
Sender : RAVI NANJUNDAPPA <[hidden email]> Chief Engineer/SRI-Bangalore-Advanced Web/Samsung Electronics
Date   : 2016-08-02 15:28 (GMT+5:30)
Title  : [Swig-user] How to cumulate %typemap(cscode) ?
 
Hi,
 
 
I have two places in my .i files where I use %typemap(cscode) on the same class.
e.g.: 
%typemap(cscode) MyClass %{
                // part #1...
%}
 
...
 
%typemap(cscode) MyClass %{
                // part #2...
%}
 
 
The problem is that they dont cumulate.  The last one wins !
 
 
I know the best would be to merge the two typemaps together but it’s impossible in my context.
(i.e.: the %typemap(cscode) is part of a macro that I call more than once on the same class !)
 
Is there a way to do it ?
 
Any help would be appreciated...
 
Thanks and Best Regards, 
N Ravi
------------------------------------------------------------------------------

_______________________________________________
Swig-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/swig-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

How to insert code into template specialization files

RAVI NANJUNDAPPA
In reply to this post by William S Fulton
Hi,

I've a requirement where in I need to insert some code snippet (C# code),  into a .cs file which has a template specialization code.

Any idea how to do that ?

Thanks and Best Regards,
N Ravi
------------------------------------------------------------------------------

_______________________________________________
Swig-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/swig-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to insert code into template specialization files

Jake
Can you be more specific about your use case?

I've used two approaches.  In some cases SWIG can already generate the code you want, 
so you can just use %extend and leave the type parameters.  Say I have a C++ type like:

template <typename T>
struct SomeType {
  T& operator[](unsigned int index);
};

Then in my interface file:

%extend SomeType {
  T GetItem(unsigned int i) { return (*$self)[i]; }
  void SetItem(unsigned int i, T value) { (*$self)[i] = value; }
}

Then these methods are part of every C# class corresponding to a concrete instantiation of 
the template (that you created using %template).

For custom C# insertion, I couldn't figure out how to get at the template parameters so I used 
a macro where I supply the type.  So to get the operator version of the above I can use:

%define %SomeType(_type)
%typemap(cscode) SomeType<_type> %{
  public $typemap(cstype, _type) this[uint index] {
    get { return GetItem(index); }
    set { SetItem(index, value); }
  }
%}
%enddef // SomeType(_type)

You use $typemap(cstype, _type) to get the C# type SWIG is using corresponding to your 
C/C++ type _type.  You can also use $csclassname to substitute the current C# class name, e.g. 
if you want to add a new constructor.  Check the docs for other available variables and functions.

The downside of this is that you have to first enumerate all the template argument types in the macro, 
then include the type definition, and then enumerate them all again in %template calls:

%SomeType(int);
%SomeType(float);
// ...

%include "SomeType.h"

%template(IntSomeType) SomeType<int>;
%template(FloatSomeType) SomeType<float>;
// ...

I would love to know how to write the typemap generically without 
relying on a bunch of macro calls if anybody can figure that out.

-Jake

On Wed, Aug 24, 2016 at 8:49 AM, RAVI NANJUNDAPPA <[hidden email]> wrote:
Hi,

I've a requirement where in I need to insert some code snippet (C# code),  into a .cs file which has a template specialization code.

Any idea how to do that ?

Thanks and Best Regards,
N Ravi
------------------------------------------------------------------------------

_______________________________________________
Swig-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/swig-user



------------------------------------------------------------------------------

_______________________________________________
Swig-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/swig-user
Loading...