[COMPLETED] Python cffi secp256k1-zkp wrapper - bounty

Anynomous · September 19, 2022, 10:41pm

Description

Python cffi secp256k1-zkp wrapper. The purpose of this wrapper is to be implemented
in the Mimblewimble implementation in Python @renzokuken is working on. The current obstacle is access to secp256k1-zkp methods. Grin uses our own fork of secp256k1-zkp library and the way it is used in Rust is via a wrapper . This bounty was created based on a request by Renzokuken the discussion in the CC meeting on the 13th of September.

Requirements

Well documented Wrapper
Well tested (test cases should be made available)
Open source, Apache License 2.0, if not possible with wrapped C code, MIT license

Resources

You may simply start grabbing definitions from

github.com

rustyrussell/secp256k1-py/blob/5bad581d959d722bf6c2df5eaa996fd4c24096aa/_cffi_build/secp256k1.h#L58-L63


      
          int secp256k1_ec_pubkey_parse(
              const secp256k1_context* ctx,
              secp256k1_pubkey* pubkey,
              const unsigned char *input,
              size_t inputlen
          );

and put them in defs.c

github.com

grinventions/secp256k1-zkp-mw/blob/004c1c86fa7da8512dd31f37d14683b4ee7f1fa2/defs.c#L58-L63


      
          int secp256k1_ec_pubkey_parse(
              const secp256k1_context* ctx,
              secp256k1_pubkey* pubkey,
              const unsigned char *input,
              size_t inputlen
          );

and then fetch the C source

git submodule init
git submodule update

and run the build

python build.py

and you will start getting parsing errors, type errors etc

Traceback (most recent call last):
  File "/Users/marek/.pyenv/versions/3.7.4/Python.framework/Versions/3.7/lib/python3.7/site-packages/cffi/cparser.py", line 336, in _parse
    ast = _get_parser().parse(fullcsource)
  File "/Users/marek/.pyenv/versions/3.7.4/Python.framework/Versions/3.7/lib/python3.7/site-packages/pycparser/c_parser.py", line 152, in parse
    debug=debuglevel)
  File "/Users/marek/.pyenv/versions/3.7.4/Python.framework/Versions/3.7/lib/python3.7/site-packages/pycparser/ply/yacc.py", line 331, in parse
    return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
  File "/Users/marek/.pyenv/versions/3.7.4/Python.framework/Versions/3.7/lib/python3.7/site-packages/pycparser/ply/yacc.py", line 1199, in parseopt_notrack
    tok = call_errorfunc(self.errorfunc, errtoken, self)
  File "/Users/marek/.pyenv/versions/3.7.4/Python.framework/Versions/3.7/lib/python3.7/site-packages/pycparser/ply/yacc.py", line 193, in call_errorfunc
    r = errorfunc(token)
  File "/Users/marek/.pyenv/versions/3.7.4/Python.framework/Versions/3.7/lib/python3.7/site-packages/pycparser/c_parser.py", line 1861, in p_error
    column=self.clex.find_tok_column(p)))
  File "/Users/marek/.pyenv/versions/3.7.4/Python.framework/Versions/3.7/lib/python3.7/site-packages/pycparser/plyparser.py", line 67, in _parse_error
    raise ParseError("%s: %s" % (coord, msg))
pycparser.plyparser.ParseError: <cdef source string>:57:5: before: secp256k1_pubkey

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "build.py", line 51, in <module>
    ffi.cdef(_source)
  File "/Users/marek/.pyenv/versions/3.7.4/Python.framework/Versions/3.7/lib/python3.7/site-packages/cffi/api.py", line 112, in cdef
    self._cdef(csource, override=override, packed=packed, pack=pack)
  File "/Users/marek/.pyenv/versions/3.7.4/Python.framework/Versions/3.7/lib/python3.7/site-packages/cffi/api.py", line 126, in _cdef
    self._parser.parse(csource, override=override, **options)
  File "/Users/marek/.pyenv/versions/3.7.4/Python.framework/Versions/3.7/lib/python3.7/site-packages/cffi/cparser.py", line 389, in parse
    self._internal_parse(csource)
  File "/Users/marek/.pyenv/versions/3.7.4/Python.framework/Versions/3.7/lib/python3.7/site-packages/cffi/cparser.py", line 394, in _internal_parse
    ast, macros, csource = self._parse(csource)
  File "/Users/marek/.pyenv/versions/3.7.4/Python.framework/Versions/3.7/lib/python3.7/site-packages/cffi/cparser.py", line 338, in _parse
    self.convert_pycparser_error(e, csource)
  File "/Users/marek/.pyenv/versions/3.7.4/Python.framework/Versions/3.7/lib/python3.7/site-packages/cffi/cparser.py", line 367, in convert_pycparser_error
    raise CDefError(msg)
cffi.CDefError: cannot parse "secp256k1_pubkey* pubkey,"
<cdef source string>:57:5: before: secp256k1_pubkey

This is just how to get started. The goal is not to copy all the methods from rustyrussell/secp256k1-py/ but to have all those as well as all the MW zkp methods from GitHub - mimblewimble/secp256k1-zkp: Fork of secp256k1-zkp for the Grin/MimbleWimble project

Amount Available

10k USD paid 90% in BTC and 10% in Grin using a 1 month average. Alternative distribution of percentage is possible upon request. 3k USD advancement is available upon showing significant progress.

Anynomous · September 19, 2022, 10:43pm

Congratulations to @walkbackgod, this bounty is locked for you . Please create a separate forum post where you give updates on the progress of this project every 2 weeks (a few bullet points will do). If we did not hear anything from you for 2 months, the bounty will become unlocked.

NicolasFlamel · December 23, 2022, 3:27pm

Now that this bounty has gone 2+ months without any activity, I’d like to request locking it to myself. I’ve already opened a PR to @renzokuken’s repo with all the deliverables that fulfill this bounty’s requirements.

github.com/grinventions/secp256k1-zkp-mw

Implements all functions from libsecp256k1-zkp

grinventions:main ← NicolasFlamel1:main

opened 05:16PM - 18 Nov 22 UTC

NicolasFlamel1

+6675 -15

This PR implements all 93 functions from [secp256k1-zkp](https://github.com/mimb…lewimble/secp256k1-zkp). This fulfills all the requirements of the [Python CFFI secp256k1-zkp wrapper bounty](https://forum.grin.mw/t/locked-python-cffi-secp256k1-zkp-wrapper-bounty/10030) established by the Grin CC. Those requirements are as follows: 1. Well documented Wrapper - The documentation for all the functions that this wrapper provides can be found in documentation.md. 2. Well tested (test cases should be made available) - Test cases for all the functions that this wrapper provides can be found in test_secp256k1_mw.py and can be tested with pytest. 3. Open source, Apache License 2.0, if not possible with wrapped C code, MIT license - The code in defs.c must be under MIT license since it's taken directly from secp256k1-zkp. - Everything else can be under any license that this project deems fit.

renzokuken · December 23, 2022, 6:30pm

Thank you for this initiative. I agree this project should back on the market. It is funny you made a PR because as I did not hear from @walkbackgod I also decided to just do it myself.

I’m sorry I missed your PR, I will review it @NicolasFlamel

renzokuken · December 26, 2022, 1:11pm

Dear all, big thank you to @NicolasFlamel for taking the initiative to complete this bounty. I just finished reviewing his PR.

All the secp256k1-zkp methods are implemented. As it goes for the wrapping work, 16% was done by @walkbackgod and the remaining 84% was done by @NicolasFlamel. Regarding the tests, entire work was done by @NicolasFlamel. Implementing tests is very difficult so I recommend to put adequate weight to that task when it comes to prize distribution.

I let the Community Council (attention @Cobragrin to include it in the agenda for the next meeting) to decide how to distribute the prize. My recommendation is to lock the bounty for @NicolasFlamel but also partially reward @walkbackgod to take into account his/her initiative and initial motivation, I recommend to set a generous deadline for @walkbackgod to pick up the prize and if he/she won’t check the message before that deadline to distribute remains to @NicolasFlamel.

The final outcome of this bounty is satisfying the requirements that were initially set. The wrapped methods build correctly with no errors, the tests run and everything is passing.

@NicolasFlamel I will comment under the PR, please let me know if you approve to have it merged before the prize gets distributed.

Thank you everyone for the great work!

NicolasFlamel · December 26, 2022, 6:13pm

With all due respect, @walkbackgod’s contributions to this project are negligible. It took me about 15 minutes to create the defs.c file since it only requires copying and pasting code from the secp256k1-zkp include headers. @walkbackgod got about 25% done with their defs.c file before they abandoned this project, and none of their work is going into the finished wrapper.

The significant majority of the time I spent working on this project went into creating the Python functions that interface with the secp256k1-zkp functions, the tests for those Python functions, and the documentation for those Python functions.

@renzokuken You can merge the PR any time

axon7 · December 26, 2022, 6:39pm

As a developer myself, having also watched this feature and tracking @walkbackgod 's progress, I feel the entire reward should go to @NicolasFlamel. The piece that wbg started, was hardly noteworthy, and if anything detracted from the progress of this project by keeping others off of it. Nick did the work. Nick should get paid.

Trinitron · December 26, 2022, 6:57pm

I also think there is a fundamental nature of bounty programs at hand, that they are paid out on completion or milestones. The original offer was for completion with optional advance for significant progress and neither was submitted.

The percentage proposed is small making this seem insignificant, but on principal I think a bounty is a bounty.

axon7 · December 26, 2022, 8:10pm

Exactly. Even though the percentage to WBG seems small, paying him partial credit without meeting any milestones will set a very bad precedent for other bounties going forward.

Anynomous · December 27, 2022, 2:01pm

Thx all, we will take this feedback with us into the next CC meeting. Thx for completing this bounty @NicolasFlamel .

renzokuken · December 27, 2022, 6:39pm

I’m sorry if I did not emphasize it sufficiently. I agree what you did is incomparably more difficult. You raised valid points and I am confident the Community Council will take them into account.

renzokuken · December 30, 2022, 5:37pm

Dear all, as @NicolasFlamel has approved merging, I proceeded and published a source distribution of secp256k1-zkp-mw.

You may install it using the following command

pip install secp256k1-zkp-mw

I tested installation and building on OSX and Linux Debian 11 and it worked. I do not have access to a modern Windows machine, so if someone has Windows and Python it would be very helpful to test the installation. If it doesn’t work, please let me know here.

NicolasFlamel · December 30, 2022, 7:02pm

Building on Windows had an issue. I’ve opened a PR that should fix it.

I also tested installing it and running all the tests with pytest on the following operating systems:

Ubuntu 22.04 x64
macOS 12.2.1 x64
Windows 10 21H2 x64 (Installed with pip install . with changes in PR)

Great job!

Memurkafasi · December 30, 2022, 8:08pm

One of the generals of the grin army.
Commander.
Hero.

Thank you Nicolas,
Thank you Renzokuken

renzokuken · January 3, 2023, 3:45pm

Hey @NicolasFlamel, I merged your PR, and publish to test pypi, unfortunately the os.remove was causing some issues, so I had to adapt a little bit (added some additional checks to remove only if empty and moving only non-existing files).

Long story short, I think solving this problem requires some deeper investigation. I enabled issues and started a discussion thread here find a way to generate the egg-info sources during the installation · Issue #3 · grinventions/secp256k1-zkp-mw · GitHub

renzokuken · January 7, 2023, 8:17pm

@Anynomous, the secp256k1-zkp-mw Python module is published, @NicolasFlamel fixed the bug described in the post before and it installs on Linux, MacOS and Windows. From my side the bounty is completed. Once the Community Council finishes the formalities regarding disbursement of the reward please don’t forget to announce it here and also maybe I would suggest to change [LOCKED] to [COMPLETED] in the title.

Anynomous · January 8, 2023, 6:27am

Thank you for taking the initiative to request this bounty and for reviewing the secp256k1-zkp-mw Python module @renzokuken. We need one more review, I think @davidtavarez is going to test the package.

Regarding the bounty distribution, it was discussed in the CC meeting on the 3rd of January on KeyBase. We want to extend our thanks and gratitude to @walkbackgod for initially taking on this bounty. However, we decided based on a) the amount of work done b) @walkbackgod not being reachable for months and c) the nature of a bounty being all or nothing unless a milestone is reached, to give 100% of the bounty to @NicolasFlamel. Ones @davidtavarez gives his we will transfer the bounty of 10k$ USD in BTC to @NicolasFlamel .

NicolasFlamel · January 8, 2023, 10:58pm

I put together an example that uses the secp256k1-zkp-mw Python module to find all the UTXOs belonging to a wallet by using its rewind hash. I hope this helps anyone who’s interested in using this module get started.

gist.github.com

https://gist.github.com/NicolasFlamel1/56becb6beb5c9cdc0f2f1d457f379f7d

scan_for_unspent_grin_outputs.py

# This demonstrates how to use the secp256k1-zkp-mw Python module, https://github.com/grinventions/secp256k1-zkp-mw, to find all the unspent Grin outputs belonging to a wallet. This code lacks any error checking. A wallet's rewind hash can be obtained by running the command `grin-wallet rewind_hash` with the Grin CLI wallet, https://github.com/mimblewimble/grin-wallet.


# Imports
from secp256k1_zkp_mw import *
from hashlib import blake2b
import requests


# Constants

This file has been truncated. show original

It’s pretty nice to have a quick way to experiment with the secp256k1-zkp functions in a language that’s simple to use now. Hopefully this attracts some new devs to the scene

davidtavarez · January 10, 2023, 8:21am

@Anynomous I’ll review it before next meeting. Thank you.

I’m excited to do it

davidtavarez · January 17, 2023, 9:33am

All tests are passing:

That’s good. The code is working as expected. I also wrote more code for testing and it is working as expected.

I tried to install the packages via pip and it wasn’t possible:

Collecting secp256k1-zkp-mw
  Using cached secp256k1-zkp-mw-0.4.2.tar.gz (72 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: cffi>=1.3.0 in d:\programs\python\python310\lib\site-packages (from secp256k1-zkp-mw) (1.15.1)
Requirement already satisfied: pycparser in d:\programs\python\python310\lib\site-packages (from cffi>=1.3.0->secp256k1-zkp-mw) (2.21)
Using legacy 'setup.py install' for secp256k1-zkp-mw, since package 'wheel' is not installed.
Installing collected packages: secp256k1-zkp-mw
  Running setup.py install for secp256k1-zkp-mw ... error
  error: subprocess-exited-with-error

  × Running setup.py install for secp256k1-zkp-mw did not run successfully.
  │ exit code: 1
  ╰─> [30 lines of output]
      running install
      running build
      running build_py
      creating build
      creating build\lib.win-amd64-3.10
      creating build\lib.win-amd64-3.10\secp256k1_zkp_mw
      copying secp256k1_zkp_mw\__init__.py -> build\lib.win-amd64-3.10\secp256k1_zkp_mw
      running build_ext
      generating cffi module 'build\\temp.win-amd64-3.10\\Release\\_secp256k1_zkp_mw.c'
      creating build\temp.win-amd64-3.10
      creating build\temp.win-amd64-3.10\Release
      building '_secp256k1_zkp_mw' extension
      creating build\temp.win-amd64-3.10\Release\Users
      creating build\temp.win-amd64-3.10\Release\Users\David
      creating build\temp.win-amd64-3.10\Release\Users\David\AppData
      creating build\temp.win-amd64-3.10\Release\Users\David\AppData\Local
      creating build\temp.win-amd64-3.10\Release\Users\David\AppData\Local\Temp
      creating build\temp.win-amd64-3.10\Release\Users\David\AppData\Local\Temp\pip-install-n5a3l5h1
      creating build\temp.win-amd64-3.10\Release\Users\David\AppData\Local\Temp\pip-install-n5a3l5h1\secp256k1-zkp-mw_b77f120f536f4642a2073c265d92c479
      creating build\temp.win-amd64-3.10\Release\Users\David\AppData\Local\Temp\pip-install-n5a3l5h1\secp256k1-zkp-mw_b77f120f536f4642a2073c265d92c479\secp256k1-zkp
      creating build\temp.win-amd64-3.10\Release\Users\David\AppData\Local\Temp\pip-install-n5a3l5h1\secp256k1-zkp-mw_b77f120f536f4642a2073c265d92c479\secp256k1-zkp\contrib
      creating build\temp.win-amd64-3.10\Release\Users\David\AppData\Local\Temp\pip-install-n5a3l5h1\secp256k1-zkp-mw_b77f120f536f4642a2073c265d92c479\secp256k1-zkp\src
      creating build\temp.win-amd64-3.10\Release\build
      creating build\temp.win-amd64-3.10\Release\build\temp.win-amd64-3.10
      creating build\temp.win-amd64-3.10\Release\build\temp.win-amd64-3.10\Release
      D:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.31.31103\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -DUSE_NUM_NONE=1 -DUSE_FIELD_INV_BUILTIN=1 -DUSE_SCALAR_INV_BUILTIN=1 -DUSE_FIELD_10X26=1 -DUSE_SCALAR_8X32=1 -DUSE_ENDOMORPHISM=1 -DENABLE_MODULE_ECDH=1 -DENABLE_MODULE_GENERATOR=1 -DENABLE_MODULE_RECOVERY=1 -DENABLE_MODULE_RANGEPROOF=1 -DENABLE_MODULE_BULLETPROOF=1 -DENABLE_MODULE_AGGSIG=1 -DENABLE_MODULE_SCHNORRSIG=1 -DENABLE_MODULE_COMMITMENT=1 -DENABLE_MODULE_WHITELIST=1 -DENABLE_MODULE_SURJECTIONPROOF=1 -IC:\Users\David\AppData\Local\Temp\pip-install-n5a3l5h1\secp256k1-zkp-mw_b77f120f536f4642a2073c265d92c479/secp256k1-zkp -IC:\Users\David\AppData\Local\Temp\pip-install-n5a3l5h1\secp256k1-zkp-mw_b77f120f536f4642a2073c265d92c479/secp256k1-zkp/src -IC:\Users\David\AppData\Local\Temp\pip-install-n5a3l5h1\secp256k1-zkp-mw_b77f120f536f4642a2073c265d92c479/secp256k1-zkp/include -ID:\Programs\Python\Python310\include -ID:\Programs\Python\Python310\Include -ID:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.31.31103\include -IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\\include\10.0.19041.0\\shared -IC:\Program Files (x86)\Windows Kits\10\\include\10.0.19041.0\\um -IC:\Program Files (x86)\Windows Kits\10\\include\10.0.19041.0\\winrt -IC:\Program Files (x86)\Windows Kits\10\\include\10.0.19041.0\\cppwinrt /TcC:\Users\David\AppData\Local\Temp\pip-install-n5a3l5h1\secp256k1-zkp-mw_b77f120f536f4642a2073c265d92c479/secp256k1-zkp/contrib/lax_der_parsing.c /Fobuild\temp.win-amd64-3.10\Release\Users\David\AppData\Local\Temp\pip-install-n5a3l5h1\secp256k1-zkp-mw_b77f120f536f4642a2073c265d92c479/secp256k1-zkp/contrib/lax_der_parsing.obj -g
      cl : Command line warning D9002 : ignoring unknown option '-g'
      lax_der_parsing.c
      C:\Users\David\AppData\Local\Temp\pip-install-n5a3l5h1\secp256k1-zkp-mw_b77f120f536f4642a2073c265d92c479\secp256k1-zkp\contrib\lax_der_parsing.c : fatal error C1083: Cannot open compiler generated file: '': Invalid argument
      error: command 'D:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.31.31103\\bin\\HostX86\\x64\\cl.exe' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> secp256k1-zkp-mw

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

I would like to suggest two things not related to the code:

Fix the pip installation.
Add a CI action on github that
a. Run the tests.
b. Build the package.

Other than that I things looks pretty good! Great job @NicolasFlamel @renzokuken

Cc @Anynomous

Topic		Replies	Views
Requesting help with writing a CFFI wrapper of our secp256k1-zkp fork Development and Technical Discussion	11	934	October 13, 2022
Grin-Bitcoin Adaptor Signature Atomic Swap update thread Development and Technical Discussion	41	5218	September 25, 2021
Mimblewimble Whitepaper V2 Research	5	1326	January 17, 2021
Eliminating finalize step revisited Research	47	681	July 25, 2023
Decoding, Decryption Grin Development and Technical Discussion	2	434	April 14, 2020

[COMPLETED] Python cffi secp256k1-zkp wrapper - bounty

Description

Requirements

Resources

Amount Available

Related Topics