SWIG › swig-user

automating C++ callback bindings for use in Python/Go

Classic

List

Threaded

7 messages Options

Brad Barfield

automating C++ callback bindings for use in Python/Go

Hi,
I am a new SWIG user and have a question about its use when dealing
with callbacks
from a C++ library. I have generated the bindings for the library APIs
successfully. However
there is one functionality that uses a callback function for async event
handling. I want
to use SWIG to create the APIs for Python and perhaps Go. So there are
a couple of issues
here, perhaps more that I don't know about yet:
1- Is there any support in SWIG that could help with this, or is
callback handling a roll
your own solution kind of thing
2- what is the general approach to this, if I want to support multiple
languages from the
library? How can the library support multiple languages above it
in such a
way as to avoid re-writing that part of the library for every new
language binding?
Thank you.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Swig-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/swig-user

Bob Hood

Re: automating C++ callback bindings for use in Python/Go

On 1/9/2017 8:37 AM, Brad Barfield wrote:

> Hi,
> I am a new SWIG user and have a question about its use when dealing
> with callbacks
> from a C++ library. I have generated the bindings for the library APIs
> successfully. However
> there is one functionality that uses a callback function for async event
> handling. I want
> to use SWIG to create the APIs for Python and perhaps Go. So there are
> a couple of issues
> here, perhaps more that I don't know about yet:
> 1- Is there any support in SWIG that could help with this, or is
> callback handling a roll
> your own solution kind of thing

Andrew Haining

Re: automating C++ callback bindings for use in Python/Go

I use the director feature to implement callbacks and it works well, it only has a limited set of language support but as far as I'm aware both python and Go are supported.

-----Original Message-----
From: Bob Hood [mailto:[hidden email]]
Sent: 09 January 2017 15:49
To: [hidden email]
Subject: Re: [Swig-user] automating C++ callback bindings for use in Python/Go

On 1/9/2017 8:37 AM, Brad Barfield wrote:

> Hi,
> I am a new SWIG user and have a question about its use when
> dealing with callbacks from a C++ library. I have generated the
> bindings for the library APIs successfully. However there is one
> functionality that uses a callback function for async event handling.
> I want to use SWIG to create the APIs for Python and perhaps Go. So
> there are a couple of issues here, perhaps more that I don't know
> about yet:
> 1- Is there any support in SWIG that could help with this, or is
> callback handling a roll
> your own solution kind of thing

I'm using SWIG 2.x, and this is how I've always had to do it because I've never seen any kind of canonical support for it. I've given some thought to an automated implementation myself, but the variations always prove to be a logistical nightmare.

I too would be interested to know if SWIG has provided a facility for this problem, even in newer releases.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Swig-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/swig-user

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
______________________________________________________________________

________________________________
Digital Barriers are world leaders in visually intelligent solutions for the global surveillance, security and safety markets, specialising in zero-latency streaming and analysis of secure video and related intelligence over wireless networks, including cellular, satellite, IP mesh and cloud. www.digitalbarriers.com Digital Barriers e-Mail Confidentiality and Disclaimer This message contains confidential information and is intended only for the individual named. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. Digital Barriers plc is a company registered in England and Wales. Registered number: 7149547. Registered office: Cargo Works, 1-2 Hatfields, London SE1 9PG, United Kingdom. For further information about Digital Barriers, please visit www.digitalbarriers.com.

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
______________________________________________________________________

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Swig-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/swig-user

Bob Hood

Re: automating C++ callback bindings for use in Python/Go

Hmm. I also use Directors rather heavily, however, in my case (I'm not
speaking for Brad here), the callbacks to which I refer are the kind you have
to provide to the host for later processing (i.e. run-time), not something
known to the compiler at compile time.

For example, let's say I need to iterate over some data contained in the host,
and the host wants to call back into my code for each item. I would have to
initiate the iteration with a "pointer" to a function for the host to invoke
(along with any attendant arguments):

typedef int (*iterate_callback)(void *object);
...
void start_iterating_over_objects(iterate_callback my_callback);

If I'm in Python, and the host is native (e.g., C++), that obviously becomes
problematic. The Python function I want to provide may not have been covered
by a Director at compile time--it could very well be a lambda.

Even providing a Director-based function becomes an issue, as it would with
any C++ class method, unless it was static. The host would have to use the
'this' pointer and an offset into the defined class to achieve the callback
effect (e.g., "(this->*offset)(...)"), which means it would have to have
compile-time knowledge of the class.

Run-time callbacks are a rather sticky subject in SWIG land, which is why I've
had to hand-code such things.While hand-coding stump functions to support
callbacks has worked for me, if there's something obvious I've missed here,
though, I would dearly love to be educated. :)

On 1/9/2017 8:55 AM, Andrew Haining wrote:

> I use the director feature to implement callbacks and it works well, it only has a limited set of language support but as far as I'm aware both python and Go are supported.
>
> -----Original Message-----
> From: Bob Hood [mailto:[hidden email]]
> Sent: 09 January 2017 15:49
> To: [hidden email]
> Subject: Re: [Swig-user] automating C++ callback bindings for use in Python/Go
>
> On 1/9/2017 8:37 AM, Brad Barfield wrote:
>> Hi,
>> I am a new SWIG user and have a question about its use when
>> dealing with callbacks from a C++ library. I have generated the
>> bindings for the library APIs successfully. However there is one
>> functionality that uses a callback function for async event handling.
>> I want to use SWIG to create the APIs for Python and perhaps Go. So
>> there are a couple of issues here, perhaps more that I don't know
>> about yet:
>> 1- Is there any support in SWIG that could help with this, or is
>> callback handling a roll
>> your own solution kind of thing
> I'm using SWIG 2.x, and this is how I've always had to do it because I've never seen any kind of canonical support for it. I've given some thought to an automated implementation myself, but the variations always prove to be a logistical nightmare.
>
> I too would be interested to know if SWIG has provided a facility for this problem, even in newer releases.
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Swig-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/swig-user
>
> ______________________________________________________________________
> This email has been scanned by the Symantec Email Security.cloud service.
> For more information please visit http://www.symanteccloud.com
> ______________________________________________________________________
>
> ________________________________
> Digital Barriers are world leaders in visually intelligent solutions for the global surveillance, security and safety markets, specialising in zero-latency streaming and analysis of secure video and related intelligence over wireless networks, including cellular, satellite, IP mesh and cloud. www.digitalbarriers.com Digital Barriers e-Mail Confidentiality and Disclaimer This message contains confidential information and is intended only for the individual named. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. Digital Barriers plc is a company registered in England and Wales. Registered number: 7149547. Registered office: Cargo Works, 1-2 Hatfields, London SE1 9PG, United Kingdom. For further information about Digital Barriers, please visit www.digitalbarriers.com.
>
> ______________________________________________________________________
> This email has been scanned by the Symantec Email Security.cloud service.
> For more information please visit http://www.symanteccloud.com
> ______________________________________________________________________
>

Daniel Schepler-5

Re: automating C++ callback bindings for use in Python/Go

I wonder if it would be possible to create a wrapper function in C++ which accepts a std::function<int(Object&)> and converts it to a C-style callback/userdata pair, then provide an input typemap which binds the std::function to a lambda or other object handling the reference counting and calling back to the Python interpreter. Of course, in the code example you gave, the API doesn't include a userdata to be given back to the callback for "closure" purposes, which would make this approach probably unworkable if that's the actual API you have to work with. But if the API does include such a thing, I think the std::function factorization would be the way I'd approach something like this.

So to give a brief idea of how this would look (I'll use Lua as an example since I'm more familiar with the C-API for Lua than for Python, but hopefully the idea should be easy to adapt):

class lua_ref { // RAII wrapper around luaL_ref/luaL_unref
private:
Lua* m_L;
int m_ref_id;
public:
lua_ref(Lua* L, int stackpos) : m_L(L), m_ref_id(LUA_NOREF) { lua_pushvalue(L, stackpos); m_ref_id = luaL_ref(L, LUA_REGISTRYINDEX); }
~lua_ref() { if (m_ref_id != LUA_NOREF) luaL_unref(m_L, LUA_REGISTRYINDEX, m_ref_id); }
lua_ref(const lua_ref& other) = delete;
lua_ref(lua_ref&& other) : m_ref_id(other.m_ref_id) { other.m_ref_id = LUA_NOREF; }
int ref_id() const { return m_ref_id; }
};

using iterate_over_objects_callback = std::function<int(Object&)>;
int iterate_over_objects_callback_wrapper(void* userdata, void* obj) {
auto the_callback = reinterpret_cast<iterate_over_objects_callback*>(userdata);
auto the_obj = reinterpret_cast<Object*>(obj);
return (*the_callback)(*the_obj);
}

void start_iterating_over_objects(iterate_over_objects_callback my_callback) {
start_iterating_over_objects(iterate_over_objects_callback_wrapper, reinterpret_cast<void*>(&my_callback));
}

%typemap(in) iterate_over_objects_callback IN {
// Take this with a huge grain of salt, even if somebody else out there *is* using Lua -
// in particular, a lot of error checking is missing.
auto lua_ref_shared = std::make_shared<lua_ref>(L, $input);
$1 = [L, lua_ref_shared=std::move(lua_ref_shared)](Object& obj) -> int {
lua_rawgeti(L, LUA_REGISTRYINDEX, lua_ref_shared->ref_id()); // Get function from reference, push to stack
SWIG_NewPointerObj(L, &obj, SWIGTYPE_p_Object, 0); // Push argument to stack
lua_pcall(L, 1, 1, 0); // Call function
int result = lua_toint(L, -1); // Get result from stack
lua_pop(L, 1); // Pop result from stack
return result;
}
}
--
Daniel Schepler
________________________________________
From: Bob Hood [[hidden email]]
Sent: Monday, January 09, 2017 8:18 AM
To: [hidden email]
Subject: Re: [Swig-user] automating C++ callback bindings for use in Python/Go

Hmm. I also use Directors rather heavily, however, in my case (I'm not
speaking for Brad here), the callbacks to which I refer are the kind you have
to provide to the host for later processing (i.e. run-time), not something
known to the compiler at compile time.

For example, let's say I need to iterate over some data contained in the host,
and the host wants to call back into my code for each item. I would have to
initiate the iteration with a "pointer" to a function for the host to invoke
(along with any attendant arguments):

typedef int (*iterate_callback)(void *object);
...
void start_iterating_over_objects(iterate_callback my_callback);

If I'm in Python, and the host is native (e.g., C++), that obviously becomes
problematic. The Python function I want to provide may not have been covered
by a Director at compile time--it could very well be a lambda.

Even providing a Director-based function becomes an issue, as it would with
any C++ class method, unless it was static. The host would have to use the
'this' pointer and an offset into the defined class to achieve the callback
effect (e.g., "(this->*offset)(...)"), which means it would have to have
compile-time knowledge of the class.

Run-time callbacks are a rather sticky subject in SWIG land, which is why I've
had to hand-code such things.While hand-coding stump functions to support
callbacks has worked for me, if there's something obvious I've missed here,
though, I would dearly love to be educated. :)

On 1/9/2017 8:55 AM, Andrew Haining wrote:

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Swig-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/swig-user

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
Swig-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/swig-user

Bob Hood

Re: automating C++ callback bindings for use in Python/Go

On 1/9/2017 12:32 PM, Daniel Schepler wrote:
> I wonder if it would be possible to create a wrapper function in C++ which accepts a std::function<int(Object&)> and converts it to a C-style callback/userdata pair, then provide an input typemap which binds the std::function to a lambda or other object handling the reference counting and calling back to the Python interpreter. Of course, in the code example you gave, the API doesn't include a userdata to be given back to the callback for "closure" purposes

It should have, and almost always does. I typed too fast. :)

I use the obligatory "userdata" argument to hold, for example, the compiled
Python instance, or a list of items.It's the only way a stump function can
operate successfully.

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
Swig-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/swig-user

Daniel Schepler-5

Re: automating C++ callback bindings for use in Python/Go

In that case, I think the factorization into two parts (C++ wrapper taking std::function, and input typemap for std::function) could have the potential to make things clearer. It could also make automation easier for the half of providing the input typemap for std::function - then all the user is left with is providing the C++ wrapper taking into account variations in the C API (which argument is the callback and which the userdata, possible lifecycle maintenance for the userdata, etc.).

For all I know, automated input typemaps for std::function could even make a good Google SoC project. :)
--
Daniel
________________________________________
From: Bob Hood [[hidden email]]
Sent: Monday, January 09, 2017 12:19 PM
To: [hidden email]
Subject: Re: [Swig-user] automating C++ callback bindings for use in Python/Go

On 1/9/2017 12:32 PM, Daniel Schepler wrote:
> I wonder if it would be possible to create a wrapper function in C++ which accepts a std::function<int(Object&)> and converts it to a C-style callback/userdata pair, then provide an input typemap which binds the std::function to a lambda or other object handling the reference counting and calling back to the Python interpreter. Of course, in the code example you gave, the API doesn't include a userdata to be given back to the callback for "closure" purposes

It should have, and almost always does. I typed too fast. :)

I use the obligatory "userdata" argument to hold, for example, the compiled
Python instance, or a list of items.It's the only way a stump function can
operate successfully.

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
Swig-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/swig-user

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
Swig-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/swig-user