Difference between revisions of "Contributing to Zynthian Development"
(Created page with "We would appreciate you to use the tools we use for developing, as it's a lot easier for coordinating the tasks, having traceability, testing, merging changes and of course, a...") |
(Typos) |
||
(123 intermediate revisions by 4 users not shown) | |||
Line 1: | Line 1: | ||
+ | == Contributing to Zynthian== | ||
+ | |||
+ | === What we expect of you === | ||
+ | |||
We would appreciate you to use the tools we use for developing, as it's a lot easier for coordinating the tasks, having traceability, testing, merging changes and of course, attributing the merit of every fix or improvement. | We would appreciate you to use the tools we use for developing, as it's a lot easier for coordinating the tasks, having traceability, testing, merging changes and of course, attributing the merit of every fix or improvement. | ||
Line 12: | Line 16: | ||
When you are a habitual contributor, you will receive write access to the repositories you master and would use development branches instead of forking/pull-requesting. | When you are a habitual contributor, you will receive write access to the repositories you master and would use development branches instead of forking/pull-requesting. | ||
− | Also, '''it's a good habit to create a task/issue associated with more relevant changes''', specially if you are a newcomer. In such a way other | + | Also, '''it's a good habit to create a task/issue associated with more relevant changes''', specially if you are a newcomer. In such a way other developers will be informed of your intention of implementing a new feature or fixing a bug and will avoid overlapping efforts, at same time allowing other developer to contribute to your task by adding useful knowledge and alternative approaches. |
+ | |||
+ | == A first time developer == | ||
+ | |||
+ | It is perfect understandable if you are approaching this for the first time, with a burning desire to solve a specific musical dilemma with code. Treat it as an opportunity to learn. It is a process that can, at first, appear overly involved and, perhaps, a little bureaucratic, but there are good reasons for the approach that's taken. It has succeeded to keep track of enormous code bases securely, sometimes in the face of actively hostile actions by others. It is this very characteristic that means that many open source projects operate that way, and may give people an insight into quite what 'open source code visible to all really' means for us. | ||
+ | Also this robustness should resolve you of fears about 'doing some damage'. This well walked path of forking a repository, and making a Pull request protects you the community from yourself in an agreed fashion. | ||
+ | |||
+ | |||
+ | === Could you tell me about the codebase? === | ||
+ | |||
+ | This project, Zynthian, is based. predominately, in the python language with various whirling bits of C & C++ doing the heavily mechanical tasks down in the bowls where life is cheap, and code turns precisely. The python is a much more starter friendly world where you define what you want to do, and this gets translated into the sort of coal one can feed into the boilers down below. | ||
+ | |||
+ | This code resides and runs within a Debian (https://www.debian.org/) based linux distribution, on a Raspberry Pi of various vintages. A zynth, without sound connections doesn't make a lot of sense so there will be some audio out device at the very least, providing an audio output that can be amplified and played to people, yourself or both. | ||
+ | |||
+ | === So what kind of thing do people do? === | ||
+ | |||
+ | It's open source. Absolutely whatever they want to do . . . | ||
+ | |||
+ | For example.. These are devices that have code written specifically for them.... | ||
+ | |||
+ | [[Supported MIDI controllers|Supported_MIDI_controllers]] | ||
+ | |||
+ | Personally I do soundscapes, church bells and cajon enhancement, and much of this built without any direct purchase from the zynthian site. The Hardware designs are up on the site if you want to make your own. You'd probably even get advise on how to do it on the forum if you ask ... | ||
+ | |||
+ | === So why might you want to write some code for a zynthian? === | ||
+ | |||
+ | Probably to do something with a particular piece of equipment that you want to customise to aid you in your sound based endeavours. | ||
+ | The output of MIDI based equipment presents it's self easily to a python code base that allows messages to be detected in an orderly fashion and then pretty much any aspect of the zynthian can be modified to obey your will. This has grown up from user dialog within the forum [https://discourse.zynthian.org/ Zynthian Forum]which is an excellent place to start with questions, suggestions and the truly crazy range of things people do with these machines. | ||
+ | |||
+ | Here's the current list of devices...[[Supported MIDI controllers|Supported_MIDI_controllers]] | ||
+ | |||
+ | === Can I do damage to MY zynthian with the code I write ? === | ||
+ | |||
+ | Yes, absolutely. | ||
+ | With power comes responsibility. | ||
+ | |||
+ | But there is an important distinction between hardware and software. | ||
+ | You can produce an inoperable zynthian with a badly placed code edit, but, you can't actively mechanically damage the machine itself by typing garbage. You simply correct the code fault and the damage its done, or at worst put in your backup ssd and things are good again. | ||
+ | |||
+ | This doesn't mean plugging the output of your 100W amp into the audio input of a zynth isn't going to damage it. It will and that's the world of hardware... | ||
+ | |||
+ | It's an important distinction. | ||
+ | |||
+ | === Can I do damage to the whole zynthian code system with the code I write ? === | ||
+ | |||
+ | No. Your code will be checked by people who control access to the core repository that you get your code from. They will not allow anything into the code base that they don't trust or that doesn't meet house rules about how stuff is written and presented. | ||
+ | |||
+ | |||
+ | === How is code of the zynthian maintained? === | ||
+ | You can log onto a zynthian itself as the root user if you have the password (opensynth), but you might well change it on a performance machine. (You may wish to avoid developing on your performance machine!) People will find it very amusing to log onto your machine pre or during performance if they can, and if you don't change the password.... | ||
+ | |||
+ | [[File:A-terminial-sshing.png|600px|center| An ssh terminal from a host computer (not necessarily yours ....]] | ||
+ | |||
+ | |||
+ | You and they (if you haven't changed the password) can, and do pretty much anything you want as it it nothing more or less than a Raspberry Pi of various vintages running an operating system. So you can edit files to your heart's content and very quickly render your zynthian unplayable if you don't know what you are doing. | ||
+ | There is an entire Linux world built around what is called a terminal, which contains a command line that you literally type commands into and press enter to get them todo things. zynthian has one of these, and there are code editors that operate entirely around the command line, allowing you to change the contents of files on the machine. | ||
+ | |||
+ | Here, there is no protection. It's your machine and you can break it as you wish, and whilst you will almost certainly get help on the discourse forum, this is not the sort of activity to indulge in twenty five minutes before your performance. It easy to become the essential technician beloved and feared in equal measure, but probably the first concept to learn is the stable branch... | ||
+ | |||
+ | |||
+ | ==== So what is a branch? ==== | ||
+ | It's a slightly different version of the code files that a program running on the zynth keeps track of, allowing you to move around between your different versions and accept of reject changes you have made. This all operates under a programme of some renown and reputation called ''git'', which was written by the same bloke who was largely responsible for the Linux code base, which is used to house the Linux code base itself, you are building on the work of giants (... and riban). | ||
+ | |||
+ | There are many, many introductions to git [https://docs.github.com/en/get-started/getting-started-with-git Getting started with git] and it has a certain reputation for complexity, but it rigorously hangs on to any code it is asked to look after even thou' this can be a mixed blessing if you are as messy as I am. But it can be operated from the command line but it's better to use an IDE (Integrated Development Environment) which makes it all ''push button'', more on this later. | ||
+ | |||
+ | Git also works peer to peer. This simply mean that rather than some of the older repository managers, every git instance running on a machine has a similar level of importance to every other, so you don't need a running central server to allow you to operate. That is a big feature that you don't appreciate until you DO work with the mega server in the middle with all it's associated limitations. | ||
+ | This doesn't mean there isn't a central repository where all the good stuff lives but it does mean that git allows you to make your own little development world where you do no damage to anybody else and you get all the control of being able to set up branches that are differing version of the code and have access to its full history whilst offline. | ||
+ | |||
+ | ==== Is Stable the best branch to use? ==== | ||
+ | |||
+ | Stable is the particular branch of code that is know good and behaves in a fashion that will match the documentation. | ||
+ | It wont be cutting edge, but it will behave itself, precisely as you expect during the performance so this is what you should be initially taking on stage. | ||
+ | |||
+ | As of 2024-11-03 Oram is the stable branch which is available from https://os.zynthian.org. | ||
+ | |||
+ | The good news is if you can get enough of a zynthian running to open up it's inbuilt webserver, forever known as ''webconf'' in these parts, you can select an option that will allow you to return to the stable state. | ||
+ | |||
+ | [[File:Zynthian software repositories.png|600px|center]] | ||
+ | |||
+ | Changing the zynthian-ui (the main zynthian code that controls the touchscreen display and the main audio and midi functionality) branch from stable to another branch and running update will get you to your desired 'branch'. Repeating in reverse to get back to stable version protects your git-committed changes but hides them and returns you to a working stable version. | ||
+ | |||
+ | This is your protection from your own misdeeds with code. In fact it is sensible to keep at least two uSD cards with a known good stable version on so you can give the zynth a brain change and keep a separate uSD card for your code projects, but if you trash a part of webconf, then you have little or no easy way onto the machine, and you will need your backup... You have been warned! | ||
+ | |||
+ | So now we know how to switch between different branches of code how do we actually set up this new branch? | ||
+ | |||
+ | Is there some magic software tool...? | ||
+ | |||
+ | ==== Who or what is Oram? ==== | ||
+ | |||
+ | On 28th September 2024 we released a new stable version: zynthian_2409 which we gave the codename "Oram" to honour [https://en.wikipedia.org/wiki/Daphne_Oram Daphne Oram], a pioneer of electronic sound and music. The ''oram'' git branch is the current stable release version which only changes when there is a ''point release'' (periodic update to deliver feature enhancement and bug fixes) or when there is a ''hotfix'' (urgent bug fixes). This is the version of zynthian that most users should use. | ||
+ | |||
+ | There is a git branch called ''vangelis'' (named after the wonderful Greek electronic music composer, [https://en.wikipedia.org/wiki/Vangelis Vangelis]) which is used to validate and test further development and bug fixes. | ||
+ | |||
+ | (If we get this right we will finally end up with a branch named after a zynthian user and it will all get needlessly messianic and meta.) | ||
+ | |||
+ | ==== So what is an IDE ?==== | ||
+ | We could operate all of this using the command line and git commands, but unless you really want to, don't. Use an IDE. | ||
+ | |||
+ | An IDE (Integrated Development Environment) is a collection of programs that work together to ease the difficulties of acquiring, writing, modifying, testing and debugging code that you write along with the possibility of many, many other functions you probably never considered. At the moment there is one that seems to be really sweeping the board with this (yes, I know, emacs users, but I worked on pycharm and intellij and I know which I prefer out of the three). | ||
+ | |||
+ | It's Microsoft's Visual Studio code, https://code.visualstudio.com/ know to friends and enemies alike as VSC (or VSCode). | ||
+ | |||
+ | [[File:Zynthian-visual-studio-code.png|600px|center]] | ||
+ | |||
+ | ==== So what is a host machine? ==== | ||
+ | |||
+ | Visual Studio Code is a considerable piece of software, Microsoft know an awful lot about writing code and they have placed a suberb, well maintained IDE into the hands of absolutely anybody with a desire to write code. (If you date back far enough to remember "a computer on every desk running Microsoft software" you might generate a rye smile at this point, but the software world is a very different place now, and Microsoft are perfectly willing to produce their IDE for every conceivable platform and operating system they come across and make it work. Anyone who plays around with Arduinos and such will find VSC's platformIO plugin is a considerable improvement on the Arduino's own IDE, and plugins like this are the secret of VSC.) The Python environment is a plugin, tools to view git repositories are a plugin, remote access components are a plugin. I haven't looked but I don't doubt there is some MIDI related plugin out there... | ||
+ | So we have a considerable chunk of code to run, which we are going to use to write a considerable chunk of code on a Raspberry Pi... Can you run VSC on a Raspberry Pi? Well yes you can, indeed I have run my VSC environment on a Pi5. But I wouldn't run zynthian and the VSC core UI code on a zynthian. There is a cleverer way. You use the remote connection plugin, to access a separate zynthian doing musical stuff. The VSC remote component installs itself onto the Pi, once you present it with the correct password (Did I mention about getting your passwords sorted out...?), and it handles many, many areas of this process that you can lose yourself into setting up yourself. I have written this process up for other environments and it was..... Involved. OK VSC advert over. | ||
+ | |||
+ | So our development environment consists of two machines, one is the zynthian itself, and the other is another computer running the VSC GI code. | ||
+ | |||
+ | This other machine is the host computer and it can be any number of bizarre, powerful, stylish, home built, clapped out, PC's and goodness knows what else, but as long as it runs VSC and has a network connection you are good to go. As I say my host machine is now a Pi5 running VSC very effectively, but your's could be MAC or a PC or goodness knows what else. This is why using a common IDE provides dividends because we don't need to explain all the different access mechanisms for all these machines. Look up how you install VSC on your host machine and do that. It's pretty easy and if anyone wants to write up specific versions then great! Just say so on the forum. | ||
+ | |||
+ | ==== How do I install VSC on my host machine? ==== | ||
+ | |||
+ | Go to [https://code.visualstudio.com/Download VSC Download page] and pick the appropriate version. For your particular Pi OS. If you are running 64 bit software pick .deb arm64, if you are on a 32 bit Pi OS then pick .deb arm32.If you are on MAC or PC then download accordingly. | ||
+ | |||
+ | [[File:Microsft-vsc-download page.png|600px|thumb|center|VSC Download Page]] | ||
+ | |||
+ | This will download the selected version and you install as appropriate to your operating system. On my Pi5 I open the download folder | ||
+ | |||
+ | <br clear=all> | ||
+ | |||
+ | [[File:File-download.png|600px|thumb|center|Pi Desktop Download Folder]] | ||
+ | |||
+ | and right click on the downloaded file and select Package Install. | ||
+ | |||
+ | <br clear=all> | ||
+ | |||
+ | It will and you will have a new icon appear in your desktop menu in the Programming section. | ||
+ | |||
+ | [[File:Visual-studio-code-menu-item.png|600px|thumb|center|Pi Desktop VSC menu item]] | ||
+ | |||
+ | Simply pressing on the icon will run Visual Studio Code, but for the moment there is a bit more to consider... | ||
+ | |||
+ | <br clear=all> | ||
+ | |||
+ | ==== Where are the zynthian source files located on a zynthian? ==== | ||
+ | |||
+ | The zynthian software exists as a set of directories (or folders if you prefer - but get used to us using the more accurate "directories") within a directory on the zynthian called... /zynthian. | ||
+ | |||
+ | [[File:Zynthian-home-directory.png|600px|thumb|center|Zynthian home directory]] | ||
+ | |||
+ | <br clear=all> | ||
+ | |||
+ | The GUI software running on the zynth that operates the touchscreen are located within the zynthian-ui directory. and there is a file called zynthian.sh. This is a shell script which isn't written in python but in a ''bash''. It is a collection of commands one runs on the terminal we mentioned earlier an allows various things to be configured, before the actual python script that does the work, which is called zynthian_main.py, is run. | ||
+ | |||
+ | [[File:Zynthian-ui-folder.png|600px|thumb|center|Zynthian-ui directory]] | ||
+ | |||
+ | One thing the shell script sets up is the debugging level which decides what level of message the zynthian generates when it encounters a logging command in the python script... | ||
+ | |||
+ | <br clear=all> | ||
+ | |||
+ | ==== What happens if zynthian crashes and how do I stop it restarting on errors ? ==== | ||
+ | |||
+ | When a zynthian starts up, most of the early functions and what gets run gets handled by a Linux tool called systemd. This organises what order things happen in and nursemaids programs, setting up procedures for what to do when programs crash or misbehave. For instance there is no point in starting up the network devices if there is no network, and this is the sort of thing systemd does on your behalf. In the zynthian case it starts up the zynthian program using the scripts we discussed in /zynthian/zynthian-ui and handles a zynthian by restarting the UI if the software crashes for some reason. This is what you want if you are performing but is decidedly irritating if you are trying to write and test code. | ||
+ | There are magic incantations to control zynthian via systemd | ||
+ | |||
+ | systemctl stop zynthian | ||
+ | systemctl start zynthian | ||
+ | systemtl status zynthian | ||
+ | |||
+ | So to stop a running zynthian you issue the first of these and this will tell systemd to ignore the restart so you can take over the running of it. | ||
+ | |||
+ | ==== How do I start a X11 windows server so I can run the code in a debugger? ==== | ||
+ | |||
+ | Once you have stopped the zynthian process, the GUI layer (X11) required to run zynthian is also stopped so you can start zynthian within the VSC debugger. To allow this to happen you have to start the X11 system to allow you to have something to connect to. | ||
+ | |||
+ | To do this in the window you typed "systemctl stop zynthian", type | ||
+ | |||
+ | X -r -s 0 | ||
+ | |||
+ | You can then start the debugger in VSC. | ||
+ | |||
+ | === How do I Debug a Zynthian? === | ||
+ | |||
+ | A careful balancing act is required here because there are a lot of entities attempting to work harmoniously here and a Pi can be a small world. | ||
+ | |||
+ | * The zynthian running on Pi, preferably as meaty as one can get. A Pi3 for instance will run out of resources with more recent engines. | ||
+ | * A Separate Computer Host from where the operation is all controlled. | ||
+ | * Some extra code loaded on the remote machine that communicates with the controlling programme on the host machine. | ||
+ | * A VNC server (A copy of the GUI image that runs in a browser) running on the zynthian to allow the screen to be examined during debugging. | ||
+ | * Some alterations to the launch.json file in .vscode to set things up nicely. | ||
+ | |||
+ | Visual Studio Code is an excellent environment for this and it fairly effortlessly sets up most of the environment for you. | ||
+ | It will need to be running on your host machine, be it Windows, Mac or Linux which could be running on a Pi. VSC runs on them all. Indeed this is being typed on a Raspberry Pi desktop machine which has VSC open and already connected to Pi4 (zynthian-rack6.local in this exercise). Running in the Linux subsystem on Chromebook also works (which is how riban had developed much of the recent zynthian code changes)! | ||
+ | |||
+ | [[File:VSC connected to zynthian.png|1200px|thumb|center]] | ||
+ | |||
+ | There are several things to notice in this screen because it is very carefully designed to be informative and make best use of screen space. | ||
+ | |||
+ | The icons on the left are worth considering. | ||
+ | Here's the [https://code.visualstudio.com/docs/getstarted/userinterface official docs...] | ||
+ | =====Explorer===== | ||
+ | Microsoft attempting to purloin a name. It's files and Directories and when configured as we intend to do here it's the files on the remote Pi4. What you edit here will change on the Pi we are examining. | ||
+ | =====Search===== | ||
+ | It's very useful to be able to perform a word or phrase search right across the entire chunk of code you are working on. This where you do that. | ||
+ | =====Source Control===== | ||
+ | It would be temping to keep all your code efforts safely backed up somewhere and if it could give you a history of what you have done in the past that would be really useful. Welcome to ''git''. It does that for you. You store the code on github.com and VSC handles the transfer of code to and from this repository for you. It also allows you to upload your code for a Pull Request (the ultimate accolade!) consideration for entering in to the zynthian code base, because you are working on a your personal fork on github of the zynthian code, aren't you? | ||
+ | =====Run & Debug===== | ||
+ | This is where we start the code examination and debugging sessions. | ||
+ | We don't start the zynthian up with the debugger running as you might suspect, it's a little bit more contrived than that. But we will explain the current (2024-09-04) dance below... | ||
+ | |||
+ | The actual controls to run the debugger are in the little window floating up at the top right of the screen. The VSC documentation [https://code.visualstudio.com/Docs/editor/debugging is here...] | ||
+ | |||
+ | =====Extensions===== | ||
+ | |||
+ | Chunks of code written by many contributors including Microsoft themselves that allow extra functionality to be loaded into VSC. The debugger itself is one such component, as is the Remote Access mechanism and an extension to work intelligently with Python code. | ||
+ | =====Remote Explorer===== | ||
+ | Access to the Remote Machine | ||
+ | |||
+ | =====Testing===== | ||
+ | Testing Module. I've not had a lot of success with this. | ||
+ | |||
+ | =====Accounts===== | ||
+ | Microsofts domain control | ||
+ | =====Manage===== | ||
+ | Admin Tasks | ||
+ | |||
+ | ====What should I do before starting VSC for debugging?==== | ||
+ | Check the VNC connection from your host machine's browser to the zynthian, by selecting VNC. | ||
+ | |||
+ | ====How do I set up VSC to use the debugger?==== | ||
+ | The zynthian has specific setup elements that allow the system to run properly. These are outlined in a file called launch.json which VSC can generate if requested but can be manually created in a directory /zynthian/zynthian-ui/.vscode | ||
+ | |||
+ | Here is an example launc.json for debugging zynthian: | ||
+ | |||
+ | { | ||
+ | // Use IntelliSense to learn about possible attributes. | ||
+ | // Hover to view descriptions of existing attributes. | ||
+ | // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387 | ||
+ | "version": "0.2.0", | ||
+ | "configurations": [ | ||
+ | { | ||
+ | "name": "Zynthian", | ||
+ | "type": "debugpy", | ||
+ | "request": "launch", | ||
+ | "program": "zynthian_main.py", | ||
+ | "python": "/zynthian/venv/bin/python3", | ||
+ | "console": "integratedTerminal", | ||
+ | "justMyCode": true, | ||
+ | "env": { | ||
+ | "DISPLAY": ":0" | ||
+ | } | ||
+ | }, | ||
+ | { | ||
+ | "name": "Zynthian Debug", | ||
+ | "type": "debugpy", | ||
+ | "request": "launch", | ||
+ | "program": "zynthian_main.py", | ||
+ | "python": "/zynthian/venv/bin/python3", | ||
+ | "console": "integratedTerminal", | ||
+ | "justMyCode": true, | ||
+ | "subProcess": true, | ||
+ | "env": { | ||
+ | "DISPLAY": ":0", | ||
+ | "ZYNTHIAN_LOG_LEVEL": "10" | ||
+ | } | ||
+ | } | ||
+ | ] | ||
+ | } | ||
+ | |||
+ | |||
+ | This allows two different configurations to be used for debugging with different levels of log reporting. | ||
+ | |||
+ | ====How do I stop the zynthian from within VSC?==== | ||
+ | Once you have a remote terminal within VSC running you type | ||
+ | systemctl stop zynthian. | ||
+ | |||
+ | This will stop the zynthian process, and your GUI Screen will disappear and the zynth wont make any sound. ( well it shouldn't). other processes like webconf will continue to work so we only need to restart the zynthian from within VSC to allow us access to VSC debugging facilities. | ||
+ | |||
+ | ====How do I re-allocate the screen?==== | ||
+ | After typing systemctl stop zynthian | ||
+ | |||
+ | type: | ||
+ | X -r -s 0 | ||
+ | |||
+ | This sorts out the graphics subsystem to work in the debugging situation. | ||
+ | |||
+ | ====How do I actually start the VSC Debugger?==== | ||
+ | Simply press the Debug Green Arrow button | ||
+ | |||
+ | <br clear=all> | ||
+ | |||
+ | ==== How do I tell the world about my contribution...? ==== | ||
+ | You put it on Github because that is where the zynthian software itself resides ... | ||
+ | |||
+ | [[File:20240120 14h10m25s grim.png|600px|thumb|center|Zynthian page at Github]] | ||
+ | |||
+ | But that is where Zynthian tells the world about its contribution and you need the good people at zynthian to approve your contribution, and they wont do that till you ask in the right way. . . | ||
+ | |||
+ | You need to submit a pull request. | ||
+ | |||
+ | ==== How do I submit a pull request...? ==== | ||
+ | This is a concept within the github environment, and to do that you need to have an account at github yourself. So you need to dance your way throu their act of anointing ([http://www.github.com www.github.com]). | ||
+ | |||
+ | Now a Pull request is not a specific git command, it's an agreement amongst mortals that the submitted code is suitable to be included in what zynthian publish to the world. | ||
+ | One of these mortals then performs a git merge to take your offering, It's called a branch, it has a name, and it contains the code you have written, and combines it into the 'testing' branch so it can be tested. | ||
+ | |||
+ | But, you say, How do I get my code up to github in the first place, and that is, unfortunately, the wrong way to think of it. | ||
+ | |||
+ | You don't put the code there, you make your own copy of the whole zynthian code base in your own github account and download it from there to your machine. | ||
+ | |||
+ | This is called a fork and it means that your submission is derived from the original git hub zynthian site and can, assuming there aren't disagreements of changes two people make (a conflict), then be merged into the code base, using the git merge command and your code is now in testing ... | ||
+ | |||
+ | Congratulations if you get this far. You have just won an open source gold star by having a PR accepted by an open source project.! | ||
+ | |||
+ | So we better write some code. | ||
+ | |||
+ | <br clear=all> | ||
+ | |||
+ | === It is 'suggested' one creates a task/issue? how do I do that ? === | ||
+ | |||
+ | "It's a good habit to create a task/issue associated with more relevant changes" | ||
+ | |||
+ | You will need a GitHub account to report issues. This is so that we know who reported it and can engage with them in its progress. | ||
+ | |||
+ | The best way to report a bug is to use the "Report Issue" button in the webconf. This creates a new issue in GitHub issue tracker, from a template with some data populated that is useful to the developers. | ||
+ | |||
+ | If you want to to this manually then, go to the zynthian issue tracker GitHub page... | ||
+ | |||
+ | [[File:Zynthian-issue-tracker-page.png|600px|thumb|center]] | ||
+ | |||
+ | <br clear=all> | ||
+ | |||
+ | Then fill out something relevant | ||
+ | |||
+ | [[File:Duo-piano feature request.png|600px|thumb|center]] | ||
+ | |||
+ | <br clear=all> | ||
+ | |||
+ | ==== So what have we gained here ? ==== | ||
+ | Well, the communal zynthian entity gains an area of interest. Even if the task, dies on the vine, and nothing more gets done, there is a possible development signposted. You also get a unique identity number that goes into the zynthian event queue. #941 in this case. This is an identifier you can add into a comment with Pull requests so the call for work and the doing of work can be related, tracked and pull requested, accepted tested and completed. | ||
+ | |||
+ | This sort of structure within a project like zynthian is a very good thing. And if precisely scratching your name into a spaceship excites you, then take this as an early buzz. | ||
+ | |||
+ | [[File:Zynthian Issues page.png|600px|center]] | ||
+ | <br clear=all> | ||
+ | |||
+ | An issue could be allocated to somebody else at some later time which means the code magically appears from someone you probably would never have met. Of course if you go ahead yourself and write it then so much the better. You move that one step closer to the "Had An Open Source PR accepted" | ||
+ | |||
+ | The GitHub environment allows these relationships to be managed and the more structured this stuff is the less intimidating it becomes... | ||
+ | |||
+ | <br clear=all> | ||
+ | |||
+ | ==== How Do I fork the zynthian GitHub repository? ==== | ||
+ | [https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/fork-a-repo The GitHub write up is here...] But in summary it's done from the appropriate zynthian page end of the process. | ||
+ | [[File:Zynthian-ui fork page.png|600px|thumb|center|Zynthian page at GitHub]] | ||
+ | |||
+ | <br clear=all> | ||
+ | |||
+ | And here are the default options which will suit nearly all circumstances. You should only strike from the testing branch. | ||
+ | |||
+ | [[File:Zynthian-ui-forking-2.png|600px|thumb|center]] | ||
+ | |||
+ | <br clear=all> | ||
+ | |||
+ | Here is the confirmation in my personal GitHub space. | ||
+ | |||
+ | [[File:Zynthian-ui-page-forked to wyleu.png|600px|thumb|center]] | ||
+ | |||
+ | <br clear=all> | ||
+ | |||
+ | ==== Can I view files on my zynthian-ui GitHub repository? ==== | ||
+ | [[File:Zynthian-ui-page-forked to wyleu.png|600px|thumb|center]] | ||
+ | |||
+ | In fact there is a very useful GitHub element that allows you to view code on gihub within a browser based tool that is nothing more or less than VSC. | ||
+ | |||
+ | It's no coincidence Microsoft bought GitHub... | ||
+ | |||
+ | To operate this feature simple select the appropriate repository home page... https://github.com/wyleu/zynthian-ui/ | ||
+ | |||
+ | <br clear=all> | ||
+ | |||
+ | and press the . key... | ||
+ | |||
+ | [[File:Zynthianwyleu-zynthian-ui-vsc.png|600px|thumb|center]] | ||
+ | |||
+ | <br clear=all> | ||
+ | |||
+ | and here in VSC in GitHub showing a ctldev file ready to be examined. | ||
+ | |||
+ | [[File:Github-vsc-editing-zynthian-ui-testing-ctrldev file.png|600px|thumb|center]] | ||
+ | |||
+ | <br clear=all> | ||
+ | |||
+ | === How Do I pass code from my Zynthian-ui github repository to my Zynthian on my desk? === | ||
+ | This is where we return to applications running on our host computer as opposed to browser based github tools . . and use Visual Studio Code(VSC) actually as an application running on our host machine... | ||
+ | |||
+ | So we start it up, | ||
+ | select Clone git repository and it's clever enough to populate the drop down if you ask it to select from GitHub. | ||
+ | |||
+ | [[File:Vsc start up.png|600px|thumb|center]] | ||
+ | |||
+ | <br clear=all> | ||
+ | |||
+ | Give it a local directory. I store my stuff under Code but it really is up to you. Try to be structural. | ||
+ | |||
+ | [[File:Git-clone-destination-folder.png|600px|thumb|center]] | ||
+ | |||
+ | <br clear=all> | ||
+ | |||
+ | And there we have a git repository located on our host machine, which is useful upto a point but isn't actually located on the zynthian . . . | ||
+ | |||
+ | [[File:Vsc-zynthian.png|600px|thumb|center]] | ||
+ | |||
+ | <br clear=all> | ||
+ | |||
+ | ==== How Do I access code on my zynthian from my host machine? ==== | ||
+ | We need to use Visual Studio Code(VSC) Remote Explorer to allow us to add on to the zynthian and install enough of an environment to allow VSC to seamlessly pass code between the environment and the desktop machine.Frankly sometimes it feels like magic, and for anyone who has read my previous attempts to do this sort of thing you spent a lot of time making sure the shaky old connections required behaved as machines are restarted and stopped and broken and goodness knows what lse. VSC relly sorts this out nicely, and it's a blessing. Indeed it is so tenacious in how it manages connections that the following image is actual a bit of a manufactured effort, but worry no it does it all very well. | ||
+ | |||
+ | |||
+ | select Connect to and enter root@zynthian.local or if you have renamed your zynthian as I tend to do to something like root@zynthian-rack2.local you put that in and enter the password for the machine. You will find yourself doing this password entry a fair bit. There are ways to make this once only per machine turning on but we will skip over that for now. | ||
+ | |||
+ | [[File:Remote-explorer.png|600px|thumb|center]] | ||
+ | |||
+ | <br clear=all> | ||
+ | |||
+ | And we now get a new item in the Remote Explorer desktop and are returned to the VSC setup page to describe where we want to locate this connection to the remote machine and it is important that you put | ||
+ | |||
+ | /home/pi/zynthian-ui in here. If you have /zynthian/zynthian-ui in her which might seem like it should work it wont for reasons to do with the long term migration to a more structured system. Don't worry it's in hand. | ||
+ | |||
+ | [[File:Remote-machine-home-directory.png|600px|thumb|center]] | ||
+ | |||
+ | <br clear=all> | ||
+ | |||
+ | So we now can examine and edit the code on the zynthian as thou' it is on our desktop... Which is kind of cool. | ||
+ | |||
+ | [[File:Remote-rack2-code display.png|600px|thumb|center]] | ||
+ | |||
+ | <br clear=all> | ||
+ | |||
+ | ==== How Do I stop being asked for my password? ==== | ||
+ | |||
+ | on your host machine type: | ||
+ | |||
+ | ssh-copy-id root@zynthian-your-machine-name.local | ||
+ | |||
+ | This will copy the key from your host computer to your remote host, where it will allow you to log on without providing the machine password each time, just a first confirmation by asking for your passphrase NOT your password. and when you need to enter the password the first time you will be asked for you pass phrase, if you set one, and since you have probably set this up for github you will have done, | ||
+ | This should reduce the chatter a fair bit, but it may still occasionally ask just out of a certain degree of paranoia.. . | ||
+ | |||
+ | <br clear=all> | ||
+ | |||
+ | ==== How Do I redirect the zynthian code in zynthian-ui to my personnal repository rather than the zynthian master I forked from ? ==== | ||
+ | |||
+ | We need to edit a specific file on the zynthian within the zynthian-ui folder. | ||
+ | It's in a hidden directory .git within the /home/pi/zynthian-ui folder so it won't show up with the ls command, use the ls -las form instead to see it.. | ||
+ | Now all this can be done from within VSC using a terminal os you have a nicely set up ssh connection to the machine all maintained by VSC. This terminal is in the lower right hand of the window and it saves you the trouble of having to set up separate connections to do basic housekeeping. | ||
+ | Because this file is actually specifically kept out of the repository it doesn't appear in the VSC code listing so we have to go and abuse it by hand, which is why we are using the terminal . . . | ||
+ | The terminal opens up in the zynthian-ui folder because that's how we configured it so all we have to do is type | ||
+ | |||
+ | cd .git | ||
+ | |||
+ | [[File:Remote-rack2-code display.png|600px|thumb|center]] | ||
+ | |||
+ | <br clear=all> | ||
+ | |||
+ | if you now type | ||
+ | ls -las | ||
+ | you should see this sort of thing ... | ||
+ | |||
+ | root@zynthian-rack2:/home/pi/zynthian-ui/.git# ls -las | ||
+ | total 92 | ||
+ | |||
+ | 4 drwxr-xr-x 8 root root 4096 Jan 20 21:46 . | ||
+ | 4 drwxr-xr-x 14 root root 4096 Jan 15 11:21 .. | ||
+ | 4 -rw-r--r-- 1 root root 16 Jan 19 15:47 COMMIT_EDITMSG | ||
+ | 4 -rw-r--r-- 1 root root 220 Jan 20 21:02 FETCH_HEAD | ||
+ | 4 -rw-r--r-- 1 root root 24 Jan 20 21:44 HEAD | ||
+ | 4 -rw-r--r-- 1 root root 41 Jan 20 21:02 ORIG_HEAD | ||
+ | 4 drwxr-xr-x 2 root root 4096 Sep 6 2020 branches | ||
+ | 4 -rw-r--r-- 1 root root 726 Jan 20 20:55 config | ||
+ | 4 -rw-r--r-- 1 root root 73 Sep 5 2020 description | ||
+ | 4 drwxr-xr-x 2 root root 4096 Sep 6 2020 hooks | ||
+ | 32 -rw-r--r-- 1 root root 32619 Jan 20 21:45 index | ||
+ | 4 drwxr-xr-x 2 root root 4096 Sep 6 2020 info | ||
+ | 4 drwxr-xr-x 3 root root 4096 Sep 6 2020 logs | ||
+ | 4 drwxr-xr-x 133 root root 4096 Jan 19 15:47 objects | ||
+ | 4 -rw-r--r-- 1 root root 334 Aug 25 00:11 packed-refs | ||
+ | 4 drwxr-xr-x 5 root root 4096 Sep 6 2020 refs | ||
+ | |||
+ | we want to modify the config file in this folder... So lets ook at this file first to see whats in it . . . | ||
+ | |||
+ | cat config | ||
+ | |||
+ | root@zynthian-rack2:/home/pi/zynthian-ui/.git# cat config | ||
+ | |||
+ | [core] | ||
+ | repositoryformatversion = 0 | ||
+ | filemode = true | ||
+ | bare = false | ||
+ | logallrefupdates = true | ||
+ | [remote "origin"] | ||
+ | url = https://github.com/zynthian/zynthian-ui.git | ||
+ | fetch = +refs/heads/*:refs/remotes/origin/* | ||
+ | [branch "master"] | ||
+ | remote = origin | ||
+ | merge = refs/heads/master | ||
+ | [branch "testing"] | ||
+ | remote = origin | ||
+ | merge = refs/heads/testing | ||
+ | [branch "staging-2209"] | ||
+ | remote = origin | ||
+ | merge = refs/heads/staging-2209 | ||
+ | [branch "stable"] | ||
+ | remote = origin | ||
+ | merge = refs/heads/stable | ||
+ | [branch "staging-2307"] | ||
+ | remote = origin | ||
+ | merge = refs/heads/staging-2307 | ||
+ | [branch "chain_manager"] | ||
+ | remote = origin | ||
+ | merge = refs/heads/chain_manager | ||
+ | |||
+ | |||
+ | We want to change the line | ||
+ | url = https://github.com/zynthian/zynthian-ui.git | ||
+ | |||
+ | to point to our own repository, which in my case is | ||
+ | |||
+ | url = https://github.com/wyleu/zynthian-ui.git | ||
+ | |||
+ | to do this we use a little command line text editor called nano... | ||
+ | so we type | ||
+ | nano config | ||
+ | |||
+ | Control-X to leave Control-O to write out the file and arrow keys move you around in the file | ||
+ | |||
+ | [[File:Nano-git-config-file=edit.png|600px|thumb|center]] | ||
+ | |||
+ | <br clear=all> | ||
+ | |||
+ | And you have now redirected your zynthian to look at your repository on github as it's ultimate origin file . . . | ||
+ | |||
+ | git pull will now cement this relationship. | ||
+ | |||
+ | <br clear=all> | ||
+ | |||
+ | ==== How Do I Start a new branch and what should I call it ? ==== | ||
+ | |||
+ | The VSC status bar at the very bottom of the screen contains a lit of useful information. | ||
+ | It shows the remote machine you are logged onto and it shows the current branch you are located on. | ||
+ | |||
+ | [[File:Making a new branch.png|600px|thumb|center]] | ||
+ | |||
+ | Indeed there are useful plugins that can be added that will show git tram lines and such which are very useful . . . | ||
+ | but if you press on the testing button in the status bar you will be presented with a dialog for adding a new branch . | ||
+ | |||
+ | Quite what to call it is upto you but remember it will live long in the github zynthian home, so choose something relevant that you can remember what it's about at some point in the future. | ||
+ | If it's too scurrilous your Pull Request will be rejected and you will be asked to change it to something more aligned with house style ... Benevolent dictators are perfectly entitled to behave like that . | ||
+ | |||
+ | In summary. | ||
+ | git doesn't forget unless you beat it very hard with a stick cos remembering what happened is it's very purpose in life. | ||
+ | |||
+ | <br clear=all> | ||
+ | |||
+ | ==== How do I switch to different branches when I'm developing? ==== | ||
+ | |||
+ | Simply use the branch name switch down on the status line at the bottom left of the vsc window. Here is is set to the branch name I created Duo-piano-device | ||
+ | |||
+ | You can also see that I have added three device files into the ctrldev directory which is the code I working on... | ||
+ | |||
+ | [[File:Git-branch-switch.png|600px|thumb|center]] | ||
+ | |||
+ | <br clear=all> | ||
+ | |||
+ | ==== Can I see a diagram of the my various zynthian various branches? ==== | ||
+ | |||
+ | Here is the git graph vsc plugin showing tramlines for this repository. You will need to install this plugin to get this. | ||
+ | |||
+ | [[File:Git-tramlines.png|600px|thumb|center]] | ||
+ | |||
+ | <br clear=all> | ||
+ | |||
+ | ==== Can I see a diagram of the various branches in the zynthian github page ? ==== | ||
+ | |||
+ | From the github.com/zynthian/zynthian-ui/ you can select insights in the top menu and then Network in the left hand vertical menu. Github will chug away for a little while having a think then show you this, which now features the duo-piano-device branch waiting for me to submit the completion of the code with some more commits and then the pull request which, if successful will result in the branch being merged into testing, and at some later date in stable . . . | ||
+ | [[File:Zynthian-ui-networks.png|600px|thumb|center]] | ||
+ | <br clear=all> | ||
+ | === How Do I start writing a ctrldev driver? === | ||
+ | Well you first have to do a lot of the process that's described here, so a quick scan of the questions and have a look at any you don't understand. | ||
+ | We assume you have vsc all set up and ready to rock and you have a debug-able zynthian attached and it runs... | ||
+ | |||
+ | ====Where do I put what files to make it work?==== | ||
+ | The ctrldev files are located in the directory /zynthian/zynthian-ui/zyngine/ctrldev in the zynthian repository zynthian-ui which is what you should have forked to get to this point. | ||
+ | It is a simple task to copy one of the existing files rename it to something that suits what you are doing and away you go.... | ||
+ | |||
+ | Well there is one other thing that needs doing to ensure the file you have added is included in the scan the zynthian makes to detect device drivers it knows about and that is to add the details of your file into the rather peculiarly named __init__.py file . . This is a default file that python likes to see in a directory to tell it what is and is not required elsewhere.... | ||
+ | As of (2024-09-09) it looks like this... | ||
+ | |||
+ | # -*- coding: utf-8 -*- | ||
+ | __all__ = [ | ||
+ | "zynthian_ctrldev_akai_apc_key25_mk2", | ||
+ | "zynthian_ctrldev_akai_midimix", | ||
+ | "zynthian_ctrldev_akai_mpk_mini_mk3", | ||
+ | "zynthian_ctrldev_duopiano", | ||
+ | "zynthian_ctrldev_korg_nanokontrol2", | ||
+ | "zynthian_ctrldev_launchkey_mini_mk3", | ||
+ | "zynthian_ctrldev_launchpad_mini", | ||
+ | "zynthian_ctrldev_launchpad_mini_mk3", | ||
+ | "zynthian_ctrldev_launchpad_pro_mk2", | ||
+ | "zynthian_ctrldev_launchpad_pro_mk3", | ||
+ | "zynthian_ctrldev_launchpad_x", | ||
+ | "zynthian_ctrldev_riband", | ||
+ | "zynthian_ctrldev_behringer_motor61" | ||
+ | ] | ||
+ | |||
+ | __all__ is a python directive to publish these modules and it means they can be seen as fully fledged members of the zynthian namespace. | ||
+ | |||
+ | This needs to match the naming of your script, which is the file zyngine/ctrldev/zynthian_ctrldev_behringer_motor61.py, which in my case I copied from the Duo_piano File... | ||
+ | |||
+ | ====How do we know we are actively running our code....?==== | ||
+ | |||
+ | We set a breakpoint in the VSC source code listing. | ||
+ | |||
+ | <br clear=all> |
Latest revision as of 08:58, 3 November 2024
1 Contributing to Zynthian
1.1 What we expect of you
We would appreciate you to use the tools we use for developing, as it's a lot easier for coordinating the tasks, having traceability, testing, merging changes and of course, attributing the merit of every fix or improvement.
We use git as versioning tool and github as repository server and issue/task tracker: Zynthian's Repository Index
If you are a punctual contributor, you should follow this procedure:
- fork the repository
- make the changes
- test it
- make a pull request
When you are a habitual contributor, you will receive write access to the repositories you master and would use development branches instead of forking/pull-requesting.
Also, it's a good habit to create a task/issue associated with more relevant changes, specially if you are a newcomer. In such a way other developers will be informed of your intention of implementing a new feature or fixing a bug and will avoid overlapping efforts, at same time allowing other developer to contribute to your task by adding useful knowledge and alternative approaches.
2 A first time developer
It is perfect understandable if you are approaching this for the first time, with a burning desire to solve a specific musical dilemma with code. Treat it as an opportunity to learn. It is a process that can, at first, appear overly involved and, perhaps, a little bureaucratic, but there are good reasons for the approach that's taken. It has succeeded to keep track of enormous code bases securely, sometimes in the face of actively hostile actions by others. It is this very characteristic that means that many open source projects operate that way, and may give people an insight into quite what 'open source code visible to all really' means for us. Also this robustness should resolve you of fears about 'doing some damage'. This well walked path of forking a repository, and making a Pull request protects you the community from yourself in an agreed fashion.
2.1 Could you tell me about the codebase?
This project, Zynthian, is based. predominately, in the python language with various whirling bits of C & C++ doing the heavily mechanical tasks down in the bowls where life is cheap, and code turns precisely. The python is a much more starter friendly world where you define what you want to do, and this gets translated into the sort of coal one can feed into the boilers down below.
This code resides and runs within a Debian (https://www.debian.org/) based linux distribution, on a Raspberry Pi of various vintages. A zynth, without sound connections doesn't make a lot of sense so there will be some audio out device at the very least, providing an audio output that can be amplified and played to people, yourself or both.
2.2 So what kind of thing do people do?
It's open source. Absolutely whatever they want to do . . .
For example.. These are devices that have code written specifically for them....
Personally I do soundscapes, church bells and cajon enhancement, and much of this built without any direct purchase from the zynthian site. The Hardware designs are up on the site if you want to make your own. You'd probably even get advise on how to do it on the forum if you ask ...
2.3 So why might you want to write some code for a zynthian?
Probably to do something with a particular piece of equipment that you want to customise to aid you in your sound based endeavours. The output of MIDI based equipment presents it's self easily to a python code base that allows messages to be detected in an orderly fashion and then pretty much any aspect of the zynthian can be modified to obey your will. This has grown up from user dialog within the forum Zynthian Forumwhich is an excellent place to start with questions, suggestions and the truly crazy range of things people do with these machines.
Here's the current list of devices...Supported_MIDI_controllers
2.4 Can I do damage to MY zynthian with the code I write ?
Yes, absolutely. With power comes responsibility.
But there is an important distinction between hardware and software. You can produce an inoperable zynthian with a badly placed code edit, but, you can't actively mechanically damage the machine itself by typing garbage. You simply correct the code fault and the damage its done, or at worst put in your backup ssd and things are good again.
This doesn't mean plugging the output of your 100W amp into the audio input of a zynth isn't going to damage it. It will and that's the world of hardware...
It's an important distinction.
2.5 Can I do damage to the whole zynthian code system with the code I write ?
No. Your code will be checked by people who control access to the core repository that you get your code from. They will not allow anything into the code base that they don't trust or that doesn't meet house rules about how stuff is written and presented.
2.6 How is code of the zynthian maintained?
You can log onto a zynthian itself as the root user if you have the password (opensynth), but you might well change it on a performance machine. (You may wish to avoid developing on your performance machine!) People will find it very amusing to log onto your machine pre or during performance if they can, and if you don't change the password....
You and they (if you haven't changed the password) can, and do pretty much anything you want as it it nothing more or less than a Raspberry Pi of various vintages running an operating system. So you can edit files to your heart's content and very quickly render your zynthian unplayable if you don't know what you are doing.
There is an entire Linux world built around what is called a terminal, which contains a command line that you literally type commands into and press enter to get them todo things. zynthian has one of these, and there are code editors that operate entirely around the command line, allowing you to change the contents of files on the machine.
Here, there is no protection. It's your machine and you can break it as you wish, and whilst you will almost certainly get help on the discourse forum, this is not the sort of activity to indulge in twenty five minutes before your performance. It easy to become the essential technician beloved and feared in equal measure, but probably the first concept to learn is the stable branch...
2.6.1 So what is a branch?
It's a slightly different version of the code files that a program running on the zynth keeps track of, allowing you to move around between your different versions and accept of reject changes you have made. This all operates under a programme of some renown and reputation called git, which was written by the same bloke who was largely responsible for the Linux code base, which is used to house the Linux code base itself, you are building on the work of giants (... and riban).
There are many, many introductions to git Getting started with git and it has a certain reputation for complexity, but it rigorously hangs on to any code it is asked to look after even thou' this can be a mixed blessing if you are as messy as I am. But it can be operated from the command line but it's better to use an IDE (Integrated Development Environment) which makes it all push button, more on this later.
Git also works peer to peer. This simply mean that rather than some of the older repository managers, every git instance running on a machine has a similar level of importance to every other, so you don't need a running central server to allow you to operate. That is a big feature that you don't appreciate until you DO work with the mega server in the middle with all it's associated limitations. This doesn't mean there isn't a central repository where all the good stuff lives but it does mean that git allows you to make your own little development world where you do no damage to anybody else and you get all the control of being able to set up branches that are differing version of the code and have access to its full history whilst offline.
2.6.2 Is Stable the best branch to use?
Stable is the particular branch of code that is know good and behaves in a fashion that will match the documentation. It wont be cutting edge, but it will behave itself, precisely as you expect during the performance so this is what you should be initially taking on stage.
As of 2024-11-03 Oram is the stable branch which is available from https://os.zynthian.org.
The good news is if you can get enough of a zynthian running to open up it's inbuilt webserver, forever known as webconf in these parts, you can select an option that will allow you to return to the stable state.
Changing the zynthian-ui (the main zynthian code that controls the touchscreen display and the main audio and midi functionality) branch from stable to another branch and running update will get you to your desired 'branch'. Repeating in reverse to get back to stable version protects your git-committed changes but hides them and returns you to a working stable version.
This is your protection from your own misdeeds with code. In fact it is sensible to keep at least two uSD cards with a known good stable version on so you can give the zynth a brain change and keep a separate uSD card for your code projects, but if you trash a part of webconf, then you have little or no easy way onto the machine, and you will need your backup... You have been warned!
So now we know how to switch between different branches of code how do we actually set up this new branch?
Is there some magic software tool...?
2.6.3 Who or what is Oram?
On 28th September 2024 we released a new stable version: zynthian_2409 which we gave the codename "Oram" to honour Daphne Oram, a pioneer of electronic sound and music. The oram git branch is the current stable release version which only changes when there is a point release (periodic update to deliver feature enhancement and bug fixes) or when there is a hotfix (urgent bug fixes). This is the version of zynthian that most users should use.
There is a git branch called vangelis (named after the wonderful Greek electronic music composer, Vangelis) which is used to validate and test further development and bug fixes.
(If we get this right we will finally end up with a branch named after a zynthian user and it will all get needlessly messianic and meta.)
2.6.4 So what is an IDE ?
We could operate all of this using the command line and git commands, but unless you really want to, don't. Use an IDE.
An IDE (Integrated Development Environment) is a collection of programs that work together to ease the difficulties of acquiring, writing, modifying, testing and debugging code that you write along with the possibility of many, many other functions you probably never considered. At the moment there is one that seems to be really sweeping the board with this (yes, I know, emacs users, but I worked on pycharm and intellij and I know which I prefer out of the three).
It's Microsoft's Visual Studio code, https://code.visualstudio.com/ know to friends and enemies alike as VSC (or VSCode).
2.6.5 So what is a host machine?
Visual Studio Code is a considerable piece of software, Microsoft know an awful lot about writing code and they have placed a suberb, well maintained IDE into the hands of absolutely anybody with a desire to write code. (If you date back far enough to remember "a computer on every desk running Microsoft software" you might generate a rye smile at this point, but the software world is a very different place now, and Microsoft are perfectly willing to produce their IDE for every conceivable platform and operating system they come across and make it work. Anyone who plays around with Arduinos and such will find VSC's platformIO plugin is a considerable improvement on the Arduino's own IDE, and plugins like this are the secret of VSC.) The Python environment is a plugin, tools to view git repositories are a plugin, remote access components are a plugin. I haven't looked but I don't doubt there is some MIDI related plugin out there... So we have a considerable chunk of code to run, which we are going to use to write a considerable chunk of code on a Raspberry Pi... Can you run VSC on a Raspberry Pi? Well yes you can, indeed I have run my VSC environment on a Pi5. But I wouldn't run zynthian and the VSC core UI code on a zynthian. There is a cleverer way. You use the remote connection plugin, to access a separate zynthian doing musical stuff. The VSC remote component installs itself onto the Pi, once you present it with the correct password (Did I mention about getting your passwords sorted out...?), and it handles many, many areas of this process that you can lose yourself into setting up yourself. I have written this process up for other environments and it was..... Involved. OK VSC advert over.
So our development environment consists of two machines, one is the zynthian itself, and the other is another computer running the VSC GI code.
This other machine is the host computer and it can be any number of bizarre, powerful, stylish, home built, clapped out, PC's and goodness knows what else, but as long as it runs VSC and has a network connection you are good to go. As I say my host machine is now a Pi5 running VSC very effectively, but your's could be MAC or a PC or goodness knows what else. This is why using a common IDE provides dividends because we don't need to explain all the different access mechanisms for all these machines. Look up how you install VSC on your host machine and do that. It's pretty easy and if anyone wants to write up specific versions then great! Just say so on the forum.
2.6.6 How do I install VSC on my host machine?
Go to VSC Download page and pick the appropriate version. For your particular Pi OS. If you are running 64 bit software pick .deb arm64, if you are on a 32 bit Pi OS then pick .deb arm32.If you are on MAC or PC then download accordingly.
This will download the selected version and you install as appropriate to your operating system. On my Pi5 I open the download folder
and right click on the downloaded file and select Package Install.
It will and you will have a new icon appear in your desktop menu in the Programming section.
Simply pressing on the icon will run Visual Studio Code, but for the moment there is a bit more to consider...
2.6.7 Where are the zynthian source files located on a zynthian?
The zynthian software exists as a set of directories (or folders if you prefer - but get used to us using the more accurate "directories") within a directory on the zynthian called... /zynthian.
The GUI software running on the zynth that operates the touchscreen are located within the zynthian-ui directory. and there is a file called zynthian.sh. This is a shell script which isn't written in python but in a bash. It is a collection of commands one runs on the terminal we mentioned earlier an allows various things to be configured, before the actual python script that does the work, which is called zynthian_main.py, is run.
One thing the shell script sets up is the debugging level which decides what level of message the zynthian generates when it encounters a logging command in the python script...
2.6.8 What happens if zynthian crashes and how do I stop it restarting on errors ?
When a zynthian starts up, most of the early functions and what gets run gets handled by a Linux tool called systemd. This organises what order things happen in and nursemaids programs, setting up procedures for what to do when programs crash or misbehave. For instance there is no point in starting up the network devices if there is no network, and this is the sort of thing systemd does on your behalf. In the zynthian case it starts up the zynthian program using the scripts we discussed in /zynthian/zynthian-ui and handles a zynthian by restarting the UI if the software crashes for some reason. This is what you want if you are performing but is decidedly irritating if you are trying to write and test code. There are magic incantations to control zynthian via systemd
systemctl stop zynthian systemctl start zynthian systemtl status zynthian
So to stop a running zynthian you issue the first of these and this will tell systemd to ignore the restart so you can take over the running of it.
2.6.9 How do I start a X11 windows server so I can run the code in a debugger?
Once you have stopped the zynthian process, the GUI layer (X11) required to run zynthian is also stopped so you can start zynthian within the VSC debugger. To allow this to happen you have to start the X11 system to allow you to have something to connect to.
To do this in the window you typed "systemctl stop zynthian", type
X -r -s 0
You can then start the debugger in VSC.
2.7 How do I Debug a Zynthian?
A careful balancing act is required here because there are a lot of entities attempting to work harmoniously here and a Pi can be a small world.
- The zynthian running on Pi, preferably as meaty as one can get. A Pi3 for instance will run out of resources with more recent engines.
- A Separate Computer Host from where the operation is all controlled.
- Some extra code loaded on the remote machine that communicates with the controlling programme on the host machine.
- A VNC server (A copy of the GUI image that runs in a browser) running on the zynthian to allow the screen to be examined during debugging.
- Some alterations to the launch.json file in .vscode to set things up nicely.
Visual Studio Code is an excellent environment for this and it fairly effortlessly sets up most of the environment for you. It will need to be running on your host machine, be it Windows, Mac or Linux which could be running on a Pi. VSC runs on them all. Indeed this is being typed on a Raspberry Pi desktop machine which has VSC open and already connected to Pi4 (zynthian-rack6.local in this exercise). Running in the Linux subsystem on Chromebook also works (which is how riban had developed much of the recent zynthian code changes)!
There are several things to notice in this screen because it is very carefully designed to be informative and make best use of screen space.
The icons on the left are worth considering. Here's the official docs...
2.7.1 Explorer
Microsoft attempting to purloin a name. It's files and Directories and when configured as we intend to do here it's the files on the remote Pi4. What you edit here will change on the Pi we are examining.
2.7.2 Search
It's very useful to be able to perform a word or phrase search right across the entire chunk of code you are working on. This where you do that.
2.7.3 Source Control
It would be temping to keep all your code efforts safely backed up somewhere and if it could give you a history of what you have done in the past that would be really useful. Welcome to git. It does that for you. You store the code on github.com and VSC handles the transfer of code to and from this repository for you. It also allows you to upload your code for a Pull Request (the ultimate accolade!) consideration for entering in to the zynthian code base, because you are working on a your personal fork on github of the zynthian code, aren't you?
2.7.4 Run & Debug
This is where we start the code examination and debugging sessions. We don't start the zynthian up with the debugger running as you might suspect, it's a little bit more contrived than that. But we will explain the current (2024-09-04) dance below...
The actual controls to run the debugger are in the little window floating up at the top right of the screen. The VSC documentation is here...
2.7.5 Extensions
Chunks of code written by many contributors including Microsoft themselves that allow extra functionality to be loaded into VSC. The debugger itself is one such component, as is the Remote Access mechanism and an extension to work intelligently with Python code.
2.7.6 Remote Explorer
Access to the Remote Machine
2.7.7 Testing
Testing Module. I've not had a lot of success with this.
2.7.8 Accounts
Microsofts domain control
2.7.9 Manage
Admin Tasks
2.7.10 What should I do before starting VSC for debugging?
Check the VNC connection from your host machine's browser to the zynthian, by selecting VNC.
2.7.11 How do I set up VSC to use the debugger?
The zynthian has specific setup elements that allow the system to run properly. These are outlined in a file called launch.json which VSC can generate if requested but can be manually created in a directory /zynthian/zynthian-ui/.vscode
Here is an example launc.json for debugging zynthian:
{
// Use IntelliSense to learn about possible attributes. // Hover to view descriptions of existing attributes. // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387 "version": "0.2.0", "configurations": [ { "name": "Zynthian", "type": "debugpy", "request": "launch", "program": "zynthian_main.py", "python": "/zynthian/venv/bin/python3", "console": "integratedTerminal", "justMyCode": true, "env": { "DISPLAY": ":0" } }, { "name": "Zynthian Debug", "type": "debugpy", "request": "launch", "program": "zynthian_main.py", "python": "/zynthian/venv/bin/python3", "console": "integratedTerminal", "justMyCode": true, "subProcess": true, "env": { "DISPLAY": ":0", "ZYNTHIAN_LOG_LEVEL": "10" } } ]
}
This allows two different configurations to be used for debugging with different levels of log reporting.
2.7.12 How do I stop the zynthian from within VSC?
Once you have a remote terminal within VSC running you type systemctl stop zynthian.
This will stop the zynthian process, and your GUI Screen will disappear and the zynth wont make any sound. ( well it shouldn't). other processes like webconf will continue to work so we only need to restart the zynthian from within VSC to allow us access to VSC debugging facilities.
2.7.13 How do I re-allocate the screen?
After typing systemctl stop zynthian
type: X -r -s 0
This sorts out the graphics subsystem to work in the debugging situation.
2.7.14 How do I actually start the VSC Debugger?
Simply press the Debug Green Arrow button
2.7.15 How do I tell the world about my contribution...?
You put it on Github because that is where the zynthian software itself resides ...
But that is where Zynthian tells the world about its contribution and you need the good people at zynthian to approve your contribution, and they wont do that till you ask in the right way. . .
You need to submit a pull request.
2.7.16 How do I submit a pull request...?
This is a concept within the github environment, and to do that you need to have an account at github yourself. So you need to dance your way throu their act of anointing (www.github.com).
Now a Pull request is not a specific git command, it's an agreement amongst mortals that the submitted code is suitable to be included in what zynthian publish to the world. One of these mortals then performs a git merge to take your offering, It's called a branch, it has a name, and it contains the code you have written, and combines it into the 'testing' branch so it can be tested.
But, you say, How do I get my code up to github in the first place, and that is, unfortunately, the wrong way to think of it.
You don't put the code there, you make your own copy of the whole zynthian code base in your own github account and download it from there to your machine.
This is called a fork and it means that your submission is derived from the original git hub zynthian site and can, assuming there aren't disagreements of changes two people make (a conflict), then be merged into the code base, using the git merge command and your code is now in testing ...
Congratulations if you get this far. You have just won an open source gold star by having a PR accepted by an open source project.!
So we better write some code.
2.8 It is 'suggested' one creates a task/issue? how do I do that ?
"It's a good habit to create a task/issue associated with more relevant changes"
You will need a GitHub account to report issues. This is so that we know who reported it and can engage with them in its progress.
The best way to report a bug is to use the "Report Issue" button in the webconf. This creates a new issue in GitHub issue tracker, from a template with some data populated that is useful to the developers.
If you want to to this manually then, go to the zynthian issue tracker GitHub page...
Then fill out something relevant
2.8.1 So what have we gained here ?
Well, the communal zynthian entity gains an area of interest. Even if the task, dies on the vine, and nothing more gets done, there is a possible development signposted. You also get a unique identity number that goes into the zynthian event queue. #941 in this case. This is an identifier you can add into a comment with Pull requests so the call for work and the doing of work can be related, tracked and pull requested, accepted tested and completed.
This sort of structure within a project like zynthian is a very good thing. And if precisely scratching your name into a spaceship excites you, then take this as an early buzz.
An issue could be allocated to somebody else at some later time which means the code magically appears from someone you probably would never have met. Of course if you go ahead yourself and write it then so much the better. You move that one step closer to the "Had An Open Source PR accepted"
The GitHub environment allows these relationships to be managed and the more structured this stuff is the less intimidating it becomes...
2.8.2 How Do I fork the zynthian GitHub repository?
The GitHub write up is here... But in summary it's done from the appropriate zynthian page end of the process.
And here are the default options which will suit nearly all circumstances. You should only strike from the testing branch.
Here is the confirmation in my personal GitHub space.
2.8.3 Can I view files on my zynthian-ui GitHub repository?
In fact there is a very useful GitHub element that allows you to view code on gihub within a browser based tool that is nothing more or less than VSC.
It's no coincidence Microsoft bought GitHub...
To operate this feature simple select the appropriate repository home page... https://github.com/wyleu/zynthian-ui/
and press the . key...
and here in VSC in GitHub showing a ctldev file ready to be examined.
2.9 How Do I pass code from my Zynthian-ui github repository to my Zynthian on my desk?
This is where we return to applications running on our host computer as opposed to browser based github tools . . and use Visual Studio Code(VSC) actually as an application running on our host machine...
So we start it up, select Clone git repository and it's clever enough to populate the drop down if you ask it to select from GitHub.
Give it a local directory. I store my stuff under Code but it really is up to you. Try to be structural.
And there we have a git repository located on our host machine, which is useful upto a point but isn't actually located on the zynthian . . .
2.9.1 How Do I access code on my zynthian from my host machine?
We need to use Visual Studio Code(VSC) Remote Explorer to allow us to add on to the zynthian and install enough of an environment to allow VSC to seamlessly pass code between the environment and the desktop machine.Frankly sometimes it feels like magic, and for anyone who has read my previous attempts to do this sort of thing you spent a lot of time making sure the shaky old connections required behaved as machines are restarted and stopped and broken and goodness knows what lse. VSC relly sorts this out nicely, and it's a blessing. Indeed it is so tenacious in how it manages connections that the following image is actual a bit of a manufactured effort, but worry no it does it all very well.
select Connect to and enter root@zynthian.local or if you have renamed your zynthian as I tend to do to something like root@zynthian-rack2.local you put that in and enter the password for the machine. You will find yourself doing this password entry a fair bit. There are ways to make this once only per machine turning on but we will skip over that for now.
And we now get a new item in the Remote Explorer desktop and are returned to the VSC setup page to describe where we want to locate this connection to the remote machine and it is important that you put
/home/pi/zynthian-ui in here. If you have /zynthian/zynthian-ui in her which might seem like it should work it wont for reasons to do with the long term migration to a more structured system. Don't worry it's in hand.
So we now can examine and edit the code on the zynthian as thou' it is on our desktop... Which is kind of cool.
2.9.2 How Do I stop being asked for my password?
on your host machine type:
ssh-copy-id root@zynthian-your-machine-name.local
This will copy the key from your host computer to your remote host, where it will allow you to log on without providing the machine password each time, just a first confirmation by asking for your passphrase NOT your password. and when you need to enter the password the first time you will be asked for you pass phrase, if you set one, and since you have probably set this up for github you will have done, This should reduce the chatter a fair bit, but it may still occasionally ask just out of a certain degree of paranoia.. .
2.9.3 How Do I redirect the zynthian code in zynthian-ui to my personnal repository rather than the zynthian master I forked from ?
We need to edit a specific file on the zynthian within the zynthian-ui folder. It's in a hidden directory .git within the /home/pi/zynthian-ui folder so it won't show up with the ls command, use the ls -las form instead to see it.. Now all this can be done from within VSC using a terminal os you have a nicely set up ssh connection to the machine all maintained by VSC. This terminal is in the lower right hand of the window and it saves you the trouble of having to set up separate connections to do basic housekeeping. Because this file is actually specifically kept out of the repository it doesn't appear in the VSC code listing so we have to go and abuse it by hand, which is why we are using the terminal . . . The terminal opens up in the zynthian-ui folder because that's how we configured it so all we have to do is type
cd .git
if you now type
ls -las
you should see this sort of thing ...
root@zynthian-rack2:/home/pi/zynthian-ui/.git# ls -las total 92
4 drwxr-xr-x 8 root root 4096 Jan 20 21:46 . 4 drwxr-xr-x 14 root root 4096 Jan 15 11:21 .. 4 -rw-r--r-- 1 root root 16 Jan 19 15:47 COMMIT_EDITMSG 4 -rw-r--r-- 1 root root 220 Jan 20 21:02 FETCH_HEAD 4 -rw-r--r-- 1 root root 24 Jan 20 21:44 HEAD 4 -rw-r--r-- 1 root root 41 Jan 20 21:02 ORIG_HEAD 4 drwxr-xr-x 2 root root 4096 Sep 6 2020 branches 4 -rw-r--r-- 1 root root 726 Jan 20 20:55 config 4 -rw-r--r-- 1 root root 73 Sep 5 2020 description 4 drwxr-xr-x 2 root root 4096 Sep 6 2020 hooks 32 -rw-r--r-- 1 root root 32619 Jan 20 21:45 index 4 drwxr-xr-x 2 root root 4096 Sep 6 2020 info 4 drwxr-xr-x 3 root root 4096 Sep 6 2020 logs 4 drwxr-xr-x 133 root root 4096 Jan 19 15:47 objects 4 -rw-r--r-- 1 root root 334 Aug 25 00:11 packed-refs 4 drwxr-xr-x 5 root root 4096 Sep 6 2020 refs
we want to modify the config file in this folder... So lets ook at this file first to see whats in it . . .
cat config
root@zynthian-rack2:/home/pi/zynthian-ui/.git# cat config
[core] repositoryformatversion = 0 filemode = true bare = false logallrefupdates = true [remote "origin"] url = https://github.com/zynthian/zynthian-ui.git fetch = +refs/heads/*:refs/remotes/origin/* [branch "master"] remote = origin merge = refs/heads/master [branch "testing"] remote = origin merge = refs/heads/testing [branch "staging-2209"] remote = origin merge = refs/heads/staging-2209 [branch "stable"] remote = origin merge = refs/heads/stable [branch "staging-2307"] remote = origin merge = refs/heads/staging-2307 [branch "chain_manager"] remote = origin merge = refs/heads/chain_manager
We want to change the line
url = https://github.com/zynthian/zynthian-ui.git
to point to our own repository, which in my case is
url = https://github.com/wyleu/zynthian-ui.git
to do this we use a little command line text editor called nano... so we type
nano config
Control-X to leave Control-O to write out the file and arrow keys move you around in the file
And you have now redirected your zynthian to look at your repository on github as it's ultimate origin file . . .
git pull will now cement this relationship.
2.9.4 How Do I Start a new branch and what should I call it ?
The VSC status bar at the very bottom of the screen contains a lit of useful information. It shows the remote machine you are logged onto and it shows the current branch you are located on.
Indeed there are useful plugins that can be added that will show git tram lines and such which are very useful . . . but if you press on the testing button in the status bar you will be presented with a dialog for adding a new branch .
Quite what to call it is upto you but remember it will live long in the github zynthian home, so choose something relevant that you can remember what it's about at some point in the future. If it's too scurrilous your Pull Request will be rejected and you will be asked to change it to something more aligned with house style ... Benevolent dictators are perfectly entitled to behave like that .
In summary. git doesn't forget unless you beat it very hard with a stick cos remembering what happened is it's very purpose in life.
2.9.5 How do I switch to different branches when I'm developing?
Simply use the branch name switch down on the status line at the bottom left of the vsc window. Here is is set to the branch name I created Duo-piano-device
You can also see that I have added three device files into the ctrldev directory which is the code I working on...
2.9.6 Can I see a diagram of the my various zynthian various branches?
Here is the git graph vsc plugin showing tramlines for this repository. You will need to install this plugin to get this.
2.9.7 Can I see a diagram of the various branches in the zynthian github page ?
From the github.com/zynthian/zynthian-ui/ you can select insights in the top menu and then Network in the left hand vertical menu. Github will chug away for a little while having a think then show you this, which now features the duo-piano-device branch waiting for me to submit the completion of the code with some more commits and then the pull request which, if successful will result in the branch being merged into testing, and at some later date in stable . . .
2.10 How Do I start writing a ctrldev driver?
Well you first have to do a lot of the process that's described here, so a quick scan of the questions and have a look at any you don't understand. We assume you have vsc all set up and ready to rock and you have a debug-able zynthian attached and it runs...
2.10.1 Where do I put what files to make it work?
The ctrldev files are located in the directory /zynthian/zynthian-ui/zyngine/ctrldev in the zynthian repository zynthian-ui which is what you should have forked to get to this point. It is a simple task to copy one of the existing files rename it to something that suits what you are doing and away you go....
Well there is one other thing that needs doing to ensure the file you have added is included in the scan the zynthian makes to detect device drivers it knows about and that is to add the details of your file into the rather peculiarly named __init__.py file . . This is a default file that python likes to see in a directory to tell it what is and is not required elsewhere.... As of (2024-09-09) it looks like this...
# -*- coding: utf-8 -*- __all__ = [ "zynthian_ctrldev_akai_apc_key25_mk2", "zynthian_ctrldev_akai_midimix", "zynthian_ctrldev_akai_mpk_mini_mk3", "zynthian_ctrldev_duopiano", "zynthian_ctrldev_korg_nanokontrol2", "zynthian_ctrldev_launchkey_mini_mk3", "zynthian_ctrldev_launchpad_mini", "zynthian_ctrldev_launchpad_mini_mk3", "zynthian_ctrldev_launchpad_pro_mk2", "zynthian_ctrldev_launchpad_pro_mk3", "zynthian_ctrldev_launchpad_x", "zynthian_ctrldev_riband", "zynthian_ctrldev_behringer_motor61" ]
__all__ is a python directive to publish these modules and it means they can be seen as fully fledged members of the zynthian namespace.
This needs to match the naming of your script, which is the file zyngine/ctrldev/zynthian_ctrldev_behringer_motor61.py, which in my case I copied from the Duo_piano File...
2.10.2 How do we know we are actively running our code....?
We set a breakpoint in the VSC source code listing.