PyAfipWs: growing up a project & surviving 15+ years in Argentina 



About:


The project PyAfipWs is a library and tools for government web-services.

I'm one of its creators & maintainer, being a python enthusiast since long time ago.


Also I'm a software developer, tech leader and now architect, and studied a Master in Free and Open Source Software (CV).


We formed a micro startup (Sistemas Ágiles: Agile Systems) to support the project and complement the community effort.


Summary:


Technology: 

  • Multiplatform library & end user tools (windows executables, interoperability & legacy interfaces, spin-offs: SOAP WS / PDF projects
  • Python 3 challenges, crypto/ssl issues
  • Useful GitHub actions for windows automation and test against external webservices.
Community: 
  • Tips and lessons learned to sustain a non trivial user base and hundreds of customers.
  • Building a 2k user group & GSoC mentorship / contributions
Resources: 

  • Commercial dual-license funding / tech support / maintenance
  • Alternatives to create sustainable open source projects for local or regional needs.

ChatGPT intro experiment:


As an alternative presentation, when asked the AI to give some information to the topic:

Q: What are the Argentina major government webservices (AFIP, ANMAT and ARBA)? Q: And what Python open source library can I use to interact with them?


The overall responses were interesting (and it mentions our library), although there are some wrong facts (strikethrough the wrong parts):
The major government webservices in Argentina are: AFIP (Administración Federal de Ingresos Públicos) - This is the federal tax agency in Argentina responsible for collecting taxes and ensuring compliance with tax laws. ANMAT (Administración Nacional de Medicamentos, Alimentos y Tecnología Médica) - This is the national regulatory agency in Argentina responsible for ensuring the safety, efficacy, and quality of drugs, medical devices, and food products. ARBA (Agencia de Recaudación de la Provincia de Buenos Aires) - This is the tax agency for the province of Buenos Aires in Argentina, responsible for collecting taxes and ensuring compliance with tax laws in the province. To interact with these webservices, you can use the Python Requests library to make HTTP requests and retrieve data from their APIs. Additionally, you may need to use Python libraries specific to each webservice to handle authentication and data parsing. For AFIP, you can use the PyAFIP library, which provides a Python interface for interacting with AFIP's SOAP web services. 😀 For ANMAT, there is no specific Python library available, but you can use the Requests library to interact with their RESTful API. 😕 For ARBA, you can use the PyARBA library, which provides a Python interface for interacting with ARBA's SOAP web services. 😱

Warning with AI hallucinations! 

PyARBA project doesn’t seem to exist (as far I could search eg. PyPI nor Google nor GitHub). 

In another question the AI also mentioned WSSRv1, a government webservice that seems to not exist either.


In fact, our library (which its complete name is PyAfipWS) supports several webservices including  AFIP, ANMAT and ARBA.


Probably it would be better to divide this project per gov agency PyARBA / PyANMAT etc. but that could be harder to maintain.


Most of them are SOAP (Simple Object Access Protocol), a mixture of HTTP + XML.

It is not really simple, except ARBA that uses something like a REST API with CSV request and XML response (so ChatGPT mixed a bit that or is referencing other webservices).


It seems the AI is translating some concepts wrongly (method names?).

Naming convention debate aside… Spanish is used in official webservice methods documentation, and some words doesn’t have a direct English translation.


Finally, when asked "Please write a code snippet in Python to generate and authorize an Electronic Invoice for Argentina AFIP webservices using PyAfip library", the response was (edited):
Sure, here is a code snippet in Python using the PyAfip library to generate and authorize an Electronic Invoice for Argentina AFIP webservices: … In this code snippet, we initialize a WSFEv1 instance with our AFIP credentials and set the invoice data. We then call the create_invoice method to generate the invoice and authorize it. Finally, we print the response.
Note that you'll need to replace the placeholders with your actual AFIP credentials and certificate paths.  

but after that there are some de


The code snippet is not that bad, the first line written (from pyafipws.wsfev1 import WSFEv1) looked promising as it mention our library and correct webservice class, but after that there are some problems. It confuses the request and response, uses the version 0 fields instead version 1, misses some tax structures as we have different VAT tax percentages, and it doesn’t point out the major result (CAE: authorization code).

Maybe ChatGPT is mixing other libraries/examples?


The correct code could be found in the project examples directory: factura_electronica.py (and there are also programs and code fragments for other programming languages and webservices).

A short fragment to illustrate and compare:

import datetime from pyafipws.wsaa import WSAA from pyafipws.wsfev1 import WSFEv1

# authentication (TA: access ticket) for testing # (creates the access request ticket and sign using certificate/private key):  
wsaa = WSAA() ta = wsaa.Autenticar( "wsfe", "reingart.crt", "reingart.key", wsdl="https://wsaahomo.afip.gov.ar/ws/services/LoginCms", debug=True, )

# electronic invoice webservice setup for testing: wsfev1 = WSFEv1() wsfev1.Token = wsaa.Token wsfev1.Sign = wsaa.Sign wsfev1.Cuit = "20267565393" ok = wsfev1.Conectar(wsdl="https://wswhomo.afip.gov.ar/wsfev1/service.asmx?WSDL") # get last invoice number: cbte_nro = int(wsfev1.CompUltimoAutorizado(1, 3)) or 0 # create the electronic invoice (header fields, simple case): wsfev1.CrearFactura( concepto=1, tipo_doc=80, nro_doc="30500010912", tipo_cbte=1, punto_vta=3, cbt_desde=cbte_nro+1, cbt_hasta=cbte_nro+1, imp_total=122.00, imp_tot_conc=0.00, imp_neto=100.00, imp_iva=21.00, imp_trib=1.00, imp_op_ex=0.00, fecha_cbte=datetime.datetime.now().strftime("%Y%m%d"), moneda_id="PES", moneda_ctz=1.000, ) # add subtotals per each VAT tax percentage (21% case): wsfev1.AgregarIva(iva_id=5, base_imp=100.00, importe=21.00) # add other taxes: wsfev1.AgregarTributo( tributo_id=99, desc = 'Impuesto Municipal', base_imp=None, alic=None, importe=1.00, ) # authorize the invoice: wsfev1.CAESolicitar() # Electronic Authorization Code (result): print("CAE", wsfev1.CAE)

Anyway it is very interesting as it give a overall help on this topic (if you know the details).

Probably we should use the Stack Overflow question/response style and provide more examples, so the information corpus about these topics increases and AI responses get more accurate.


Quick Intro (real):


After the ChatGPT experiment, that was interesting but probably there is no enought information for more correct responses (as documentation is in Spanish and very limited).


Let's do a traditional quick introduction to these topics.


AFIP is the country's main tax agency, like the "I.R.S." in the US (Internal revenue service)..


Note that every invoice in Argentina must be sent to the government to get approval (including total amount, taxes plus item description, quantities and unit prices in some cases), or use a special printer with government software (fiscal controller).


There are special regulations to use invoices for tax returns (industrial fiscal bonus), foreign trade commerce (exports), tourism or using credit invoices as checking-account to pay other invoices.

There is a tax-payers registry to validate customers' VAT numbers and status. 

And received invoices should be verified using a webservices to validate the amounts for tax porpouses.


Also the primary sector is important in Argentina, so sales and shipments of cereal crops, flour, cattle, meat or milk production and derivatives should be informed to  Afip too, using several Agriculture or Livestock webservices.


Customs and foreign exchange are also complicated, for example in the past needing authorization to buy or sell US dollars (via web service for companies/banks or online government website for taxpayers).


Other related web services supported by the library are: 


ARBA: Buenos Aires province tax agency

  • Freight transport Shipments code (COT), also for Córdoba & Santa Fe provinces.
  • Gross income tax withholdings (IIBB).

National Traceability System web services: to track movements of

  • Medicines and medical equipment (ANMAT TrazaMed/ProdMed)
  • Controlled substances (SEDRONAR/RENPRE TrazaRenpre)
  • National Health and Food Quality (SENASA: TrazaVet/TrazaFito)

No official python example, even in 2023!

The official  AFIP Government website has general documentation about its web services: afip.gob.ar/ws
The architecture is divided in two stages:

The client calls WSAA (authentication and authorization webservice) to get a Token and Sign.

A custom digital X509 certificate is used and requires signing a CMS (Cryptographic Message Syntax format). It is basically like a "Signed Multipart Email" (S/MIME). The result is an access ticket XML.

Then the client uses that access ticket to consume the WSN (business webservice), sending the Token and Sign along business data. It can do several operations as long as the access ticket is valid (not expired).
Business web services are Electronic Invoice Version 0, Version 1, Fiscal bonus, Exports, Tourism, etc..

Developer Manuals are agnostic PDF with XML Request / Response templates and some minimal examples with test data (in the best cases).

There are some programming examples for WSAA, sadly only for JAVA PH C#, VB.NET and PowerShell.
Note that they don't have any official programming tech support.

And some use cases are tricky or there are usage limits (for example, 5 minutes to request a new token, or non trivial error recovery procedures) that could be difficult to handle for newcomers.

The full catalog includes more than 40 web services with many versions and regulations!. 

They cover from invoicing to registry, agriculture/livestock, tax forms filing, payroll, customs, and more specific use cases.


For these webservices there are some some official examples for PHP / Java .NET, but they are not supported and no Software Development Kit exists.

And no, no python example, even in 2023 😞

Official examples were a bit complex to setup (specially for legacy systems) and have many dependencies.

There are now some command-line utilities to use WSAA and get an access ticket (token/sign) using powershell.

Note that official docs point to OpenSSL / SoapUI and WireShark to implement / test / debug this web services (that could not be easy specially for newcomers)

Motivation

This library surges to build a community alternative as there were no free / open source Software Development Kit.

A main goal of this project is to be easy to use, specially for developers of other languages.
Requirements should be easy to install and eventually maintain or adapt.

For example Python SOAP library at that time had some memory consumption issues and were not entirely compatible with the different XML dialects for Java or .NET (we tried but we couldn't adapt them).

Also multi-platform and interoperability were a must, as many users had legacy operating systems or applications that cannot easily be extended to add JAVA or .NET requirements.

This include old Windows versions (XP), Unix flavors (some no longer actively supported), and programming languages that do not have even a networking stack or XML / crypto services (COBOL, Visual Basic, Visual FoxPro, Pascal and dBase derivatives, etc.).

Python offered these features, and also made possible desktop applications (PyRece / PyFactura): 
 

Finally, user experience was important to bring new developers to the python world without hassle.
Single file installers with no runtime or other dependencies were implemented.

The origins:

The project dates back to 2008, when we exchanged some mails in the Python Argentina users group. 
We joined efforts and started working in a common library.

Then freelance portals were used to offer the library to developers needing to implement electronic invoice for legacy applications in Cobol / FoxPro (DBF) and similar.
There were also diffusion in forums & user groups of other languages (VB/VFP).

Eventually even some Java, .NET and PHP users also started to look at this library, as we gave tech support (remember no official programming support was given from the government, just XML error issues)

pywin32 is used to expose Python Objects to other modern languages (VB / VFP support CreateObject for automation) similar to an OCX control (COM: Component Object Model).
py2exe is used to build EXE / DLL to host the object servers and command line tools.

Current status

The road so far includes around 25 webservices supported, 5 Electronic Invoice webservices (in 2 variants), 8 agriculture webservices and several other related agencies and systems.

We added PDF generation, with barcode and QR, plus mailing features.

File interchange formats include universal fixed-width text format (COBOL), XML, DBF tables, CSV/XLS spreadsheets and JSON.

Additional command line tools were developed to create access tickets, authorize, retrieve and verify invoices, bulk processing (reading invoices from databases), etc. 

Also two Visual desktop applications were created to overcome some limitations and issues of government old windows utilities and official website. 

They are oriented to non-programmers (for example, accountants or small shop owners) with bulk processing features and customization/storage capabilities.

We were invited to several meetings in AFIP offices and to participate in the related group of Argentine Software Industry Chamber.

The User's Group

We created pyafipws@googlegroups.com that over time attracted 2k members… (in comparison, Python Argentina telegram group has 2.4k members)

It works as a public forum and is very active and used for several topics:
Many users helps other users, contributing error corrections and minor examples.

Topics exceeding this python library are included too:
  • Announcements & discussions about AFIP government news and regulations.
  • AFIP downtime and general scalability issues .
There is some moderation to avoid political discussion and non-free software (spam).

Sustainability

To sustain development and maintenance we opted for a dual license model.
Licensed under GPLv3 for free software projects, and a commercial exception for proprietary software.
Now moving to LGPL and looking alternatives ways of funding; 
GPL is great to raise awareness but there are grey areas and could be more hard to manage for small projects (for example contributor agreements)

We offer commercial technical support with a small update fee (around 50 USD on average). 
It is mainly for administrative, installation issues and first steps.

This helps to do basic maintenance, and we have more advanced plans for releases of new webservices or major changes. 

But there are periods without activity nor significant income (updates are optional), so major refactors are difficult.
Only large companies are willing to pay monthly/yearly support plans, mainly to cover operation, regular maintenance and minor enhancements.

This funding also was used initially to pay for AdWords campaigns in search engines.
It complemented SEO and allowed new users to find the project when it was not widely known.

Public forum also helped to communicate and generate synergy with users helping each other (even non commercial free riders).

For SEO, a landing page was published with a lot of information and details on regulations.
Also the full documentation is freely available.

Over years some links and resources could get outdated, so special care must be taken to keep it updated (that is also included in the tech support fee)

Documentation:

Example documentation page (originally using trac), with a menu listing all webservices and tools (left).

Contents include all AFIP government regulations, downloads / installation instructions, examples for different programming languages, pseudocode and sample files, use cases, common error and exceptions, webservices codes and parameter tables, etc.

Python 3

Crossing the chasm?

Python 3 porting was not trivial and it depended on several third party libraries.

Some, like m2crypto (openssl) was eventually ported but more difficult to install (replaced by cryptography)

Others, like py2exe, were not ported initially or had limited functionality (until recently it didn’t supported some features we needed).

Specially on Windows we faced issues with SSL protocol and support proxy (for example pycurl)

We had to opt to keep a quick reaction to government changes and fixing bugs.
User base is mostly in python 2, and they wouldn’t pay support just to port.

We still need to keep compatibility with legacy users (old Windows or Unix installations)

Spin-offs

We started with simple modules for SOAP and PDF that over time become independent libraries.

SOAP code was moved out to its own project (PySimpleSOAP), and eventually many contributors were interested in submitting changes and enhancements.
It was later moved from a personal project to an organization and had active development, but over time interest stalled. 
Some changes introduced by other maintainers were incompatible with our PyAfipWs library (or were untested in our scenarios), so at the end we’re using an older branch with custom changes (see py311).

PyFPDF was born from a port of the fpdf PHP library (intended for simple generation of PDF), with many enhancements and also eventually attracted many contributors. 
The project was forked to fpdf2, python 2 support was dropped and some changes are incompatible with upstream API (see #111).
So we cannot use the latest branch now, hence we still keep working on our older branch.

It is a bit strange to step down for your own libraries, but also it is understandable that the community needs to advance (especially if changes cannot be reviewed in a timely basis, or security/compatibility concerns arise).

NOTE that sometimes priorities change, and original maintainers can lack enough time to work on the project (in my case, I had 3 jobs in some periods, so open source dedication was limited).

Also, it could be needed to say no and reject contributions if those cannot be reviewed or tested, or introduce breaking changes or other issues.

Integrations:

This could be a major concern especially if the library is used in larger projects (in our case, Argentina localizations for OpenERP/Odoo and Tryton), as it could introduce bugs or security issues to them.

Open Source never dies 😉
Low maintenance mode is good

If a project doesn't receives a lot of changes, it could just mean that it has no serious bugs or new features are not needed. And it can be forked if needed...

A humble recommendation would be to accept new contributions but in a new branch (2.0 “next” or unstable), and use versioning to pin stable 1.0 branch.
When the new 2.0 branch becomes stable, reviewed and security/compatibility issues solved, you can easily change to the new branch. 
This balance could avoid friction and unnecessary forks, but will not be completely free/easy. Anyway it could be better that having unnecesary forks or different version in different projects.

Modern Era:

In the search for new contributors, we participated in Google Summer of Code 2019, 2021 and probably in 2023 (see PyAr ideas wiki page), under the Python Software Foundation umbrella.

The students get a grant to learn and develop (around 1500 USD). 
The PSF receives 500 USD that later are used in other grants programs.

In our case, the students became mentors for the project, and in general this program helped us a lot to tackle big ideas / refactors like Python 3 port, dependency updates, testing improvements (VCR) and stabilization.

In the last 5 years we moved from Travis to GitHub Actions.
This helped to run the test with different python versions and operating systems (linux/windows)
Now windows installers can be almost built in C.I., and it also can be tested in a fresh installation with some basic automation tests.

This new Continuous Integration pipeline helps to review and approve pull request fasters, as they can be tested automatically in all scenarios.

GitHub actions:

Python package workflow has a build job for several python versions.
The job has the standard steps to checkout, install dependencies, configure and run unit tests with pytest. 
A test report is built and coverage report is uploaded to artifacts.

The other Windows Installer workflow is run for 32 and 64 bits.
It has several jobs (compile, test and pre-release).
In the compile phase it builds the project and convert to an executable using py2exe.
Then the test phase downloads the certificates (in secrets to avoid conflicts).
After that it runs the executables for simple checks.
And finally it registers the components and run some automation tests with pester (powershell).
The final pre-release job upload the artifacts (dist archive with executables and python libraries)

Testing

Originally we had unit tests and end 2 end tests that had to be run manually.
Some test were difficult to run as they depend on  AFIP webservers that can go down / raise timeouts or have limitations (certificate cannot be used once each 5 min).

We use VCR to record requests and replay responses, stored in yaml “cassettes”, so flakiness due external services is reduced.
In tandem, freezegun is used to keep the request reproducible (immutable date/time).

For End 2 End tests we use pester PowerShell tests. 
These tests are more difficult to mock as they are black-box processes, but a fake webservice simulator could be used in the future.

In the test folder we have the cassettes folder with the recorded request/response.
For example the login yaml has the webservice URL endpoint and the XML request & response, with all the flags.
This is later used in test wsaa.py via the @pytest.vcr decorator.
In test wsfev1.py we enable the pytest.vcr and pytest.freezegun plugin for all the tests.

Finally, the powershell folder has the pester tests using windows automation.
For test wsaa.ps1 there are some basic test to generate the xml access ticket.
Then in test wsfe.ps1 we have a basic test for electronic invoice generation, checking the happy path and some corner cases / exceptions. 

References:

Thank you for reading!
Fell free to contact me if you have any doubt

This is a transcription (slides) with extended details of the talk presented for the PyCon US 2023 Maintainers Summit
Licencia y Aviso Legal
Licencia y Aviso Legal

CC BY-SACopyright (C) 2009 - 2013 Mariano Reingart reingart@gmail.com - Visite el sitio http://reingart.blogspot.com/ para novedades. Salvo indicación contraria, el texto de esta página está disponible bajo la Licencia Creative Commons Atribución Compartir Igual 3.0 Unported; podrían ser aplicables cláusulas adicionales.

Se solicita mantener el aviso de copyright (enlazando al texto original), compartir las obras derivadas bajo la misma licencia, y si es, posible comunicando al autor cualquier mejora. No utilizar logotipos ni marcas sin la debida autorización de sus titulares.

Toda información es proporcionada a Titulo Informativo y se entrega como está, sin garantías explícitas ni implicitas de ningún tipo, incluyendo sin limitación, pérdida de ganancias, interrupción de negocios, pérdida de programas u otros datos en sistemas de computación o cualquier otro reclamo. Al usarla acepta hacerlo bajo su propia responsabilidad, conociendo la normativa y reglamentaciones existentes. En caso de controversias respecto del presente, usted acepta presentarlas y resolverlas exclusivamente en la jurisdicción de los Tribunales Ordinarios Civiles y Comerciales del Departamento Judicial de Morón, Provincia de Buenos Aires, Argentina, renunciando expresamente a cualquier otro fuero que pudiere corresponder.


Enlaces
Laboral
Laboral
Laboral
Grupo de desarrolladores independientes
Docente en
Docente en
Docente en
Instituto Superior Blaise Pascal
Diplomatura
Diplomatura
Diplomatura
Universidad del Este
Proyecto PyAfipWs
Proyecto PyAfipWs
Proyecto PyAfipWs
Interfáz libre a los servicios web de AFIP
Etiquetas
Datos personales
Datos personales
Suscribirse
Suscribirse
Archivo del blog
Cargando