Tuesday, November 30, 2010

darcs weekly news #80

News and discussions

  1. Alexey Levan is our new Windows packager, and has put online Windows binaries of Darcs 2.5:
  2. We posted two roadmap blogs about Darcs 2.8:

Issues resolved in the last week (0)

Patches applied in the last week (1)

See darcs wiki entry for details.
Posted by at 2 comments:
Labels:

Coming in Darcs 2.8: new features

In the previous post we talked about Darcs 2.8's read-only support for old-fashioned (OF) repositories. This reduced support will help us simplify the code base and focus on new features and optimizations for the Hashed repositories, that are now the recommended way to use Darcs.

New features

Let us review the features that were not in 2.5 and that we are working on now:
  • rebase: this is a long-wanted Git feature that may be merged into Darcs 2.8. It is currently maintained on a separate branch. We will be confident to include it in the 2.8 release if we are sure we get the UI right by then. We already had some feedback but if you want to try it please go ahead and tell us what you think.
  • repositories without working copies: this is still not accepted in HEAD but we are confident we can have it on time.
Let us now consider an improvement that is already present in Darcs HEAD and is thus guaranteed to appear in 2.8: packs. Basically this optimization makes getting a repository over HTTP much faster.

What happens with darcs get

Let us come back to the darcs get command and see why fetching Hashed repositories is faster than OF (see also the previous post for more information on OF repositories). A Darcs repository contains (among other things):
  • the patches
  • a pristine cache, that is the "sum of patches"
Here is what Darcs does when getting an OF repository:
  1. get patch 0, get patch 1, ..., get patch n
  2. build the local pristine and working directory trees
Hence to start hacking on a project hosted like this, you need to wait as long as the history of the project is large. This is only bearable for small histories. For instance, the Xmonad repository (>1100 patches) can take up to 7 minutes to be retrieved on an ADSL line, while the repositoy itself is only 5 MBytes big.

On the other hand, when getting a Hashed repository, Darcs does:
  1. get pristine file 0, get pristine file 1, ..., get pristine filem
  2. get patch n, get patch n-1, ... , get patch 0
  3. build local working directory tree
Step 2. can be skipped with the --lazy flag, or can be interrupted by hitting CTRL-C. This is fine, since the new local repo has the address of the remote repo and Darcs can fetch patches on demand. Hence, the wait can be reduced a lot, and dowloading with --lazy is as long as the working copy is large. The Xmonad repository has about 30 pristine files, weighting less than 200 KBytes, so getting those would only be a matter of seconds.

By the way, getting a Darcs repo with --lazy is strictly more powerful than doing a checkout of a Subversion repository: even though you don't have patches locally, you have the high-level history of the project (i.e., you can use darcs changes offline). However, looking into the patch contents (darcs changes -v) does require you to access the remote repository.

Darcs 2.8 will contain a new optimization called packs. Running the command darcs optimize --http in a repository will store a pack of the pristine and a pack of the patches inside of it. Packs are basically tar.gz archives. Darcs will detect these packs when doing a get and act as follows:
  1. get pristine pack
  2. get missing pristine files
  3. get patches pack
  4. get missing patches
  5. build working directory tree
Steps 1. and 3. consist in transferring a single file via HTTP, which is much faster than transferring many little files. Steps 3. and 4. are skipped if the flag --lazy is used.

How much faster is this? Here are the figures we got on the Darcs repository  (>8000 patches, >600 pristine files):
  • full get without packs: 40 minutes
  • full get with packs: 3 minutes
  • lazy get without packs: 20 seconds
  • lazy get with packs: 11 seconds
You need to run darcs optimize --http manually on the public repository of your project from time to time, as this will not happen automatically. The best moment to do that is after pushing a tag, since this enables people who use get --tag X (with X being the last tag) to take advantage of the optimization.

Conclusion

This is probably not going to be the only new features that we will try to fit into the 2.8 release, so when we have more we will let you know. Before May 2011 we are going to try to release feature-based alpha  versions of Darcs. They will be called Darcs 2.7.x and will be for users  who want to try out bleeding-edge features and give us feedback.

If you can't wait, you can simply build the current Darcs HEAD with darcs get --lazy http://darcs.net and cabal update && cabal install -f-library inside of the obtained directory.

EDIT: removed darcs-fastconvert from the list since as of now this is not a supported feature of Darcs.
Posted by at 4 comments:

Tuesday, November 23, 2010

Coming in Darcs 2.8: read-only support for old-fashioned repositories

A month ago, we released Darcs 2.5. This version brought performance improvements and a few nice features like trackdown --bisect and UTF8 patch metadata. Since we follow a time-based release schedule, Darcs 2.8, the next major release, will be released in May 2011, and we are at the beginning of the work that will be part of it. So what are we working on now?

One of the hot topics of Darcs 2.8 will be repository formats. As of now, Darcs can work with two kinds of repositories: the old-fashioned (OF) ones and the Hashed ones. OF has been around since the very first Darcs, while the Hashed format has been introduced in Darcs 2.0 in April 2008.

Why Hashed

The reasons why the Hashed format was introduced were robustness and performance.

To understand robustness, one has to know that darcs repositories contains patches (of course), but also a "pristine cache" which is the latest recorded state of the files stored in the repository. It can be rebuilt from the patches of the repository, so it is not really mandatory in theory, however it makes some commands run in reasonable time. The command whatsnew, for instance, show the current unrecorded modifications in your working copy, and works by comparing your working copy with the pristine.

In OF repositories, the pristine directory is simply a plain filesystem tree. The consequences are that if an external program (say Subversion or Unison) adds files to that directory, Darcs will believe they really belong to the pristine cache! Hence, the command whatsnew would tell that you are missing some file in your working copy, and the command record would record a wrong patch! More generally, OF repositories can not be really checked for integrity: Darcs can not know you modified a pristine file by hand.

In Hashed repositories, the pristine is a directory containing files named by the SHA1 hash of their contents, and there is a special file that tells the hash of the root directory of the pristine. Hence, adding bogus files to the pristine directory is harmless: they will never be read. So bring on all your Subversions, Unisons and Dropboxes: Darcs can not longer be tricked.

There is another downside of OF: getting them via HTTP is very slow. Indeed, they contain no list of pristine files. You can not directly copy the remote pristine files in order to get a working copy. So you need to get all the patches, and then locally rebuild a pristine and a working copy, making the delay until getting a working copy linear in the history of the project. On the other hand, the latest working copy of a Hashed repo can be obtained with darcs get --lazy, which does not need to retrieve the history of the project.

The developers are not amused

Those are downsides of OF for users. But we, as developers, are also not very pleased with some parts of the Darcs codebase. The code that handles repositories is not really great and lacks modularity. Moreover, nobody is really motivated to maintain code for OF, since we know Hashed repositories are better and there is ongoing work in order to make Darcs more performant with them. For instance, the work done in Summer of Code 2009 has then been incorporated in Darcs 2.4 and has boosted performance of Darcs a lot. In Darcs 2.5, performance of commands like record and pull have been substantially improved. None of these changes concern OF repositories.

Hence we have decided to stop struggling with old code, and from Darcs 2.8, only read-only access will be provided for OF repositories. This means you will still be able to use the commands get, pull  and send to interact with remote OF repositories. On the other hand writing in an OF repository with record, pull or apply will no longer work, and commands that rely on the pristine files, like whatsnew, diff or dist, will also fail. We will make sure the user interface will remain clear and  helpful in those cases.

For us, this is a step further towards simplify the code base and make it generally more modular. This will help us focussing on performance improvement for the Hashed format and modern features for the Darcs client without having to think about OF maintenance.

Should it concern you?

Switching to Hashed repositories has a drawback: Darcs 1 binaries do not know how to interact with them. If you manage a project whose source code is hosted in a Darcs repository, then you should ensure that all contributors use a Darcs 2 binary (darcs --version).

If you or someone in your project use a Darcs 1 binary, you should check whether it is possible for you to upgrade to Darcs 2.0 or greater. Please consult http://wiki.darcs.net/Binaries for more information and http://wiki.darcs.net/DarcsTwo for converting from OF to Hashed.

For the record, Darcs 2.0 was released in April 2008 and is now adopted in most current versions of the major Linux distributions. The release of Darcs 2.8 is planned for May 2011, so no problematic situation due to OF deprecation will happen before then.

    Consolidating the move

    We know that a few shortcomings remain with hashed repositories. For instance darcs add is noticeably slower on big Hashed repositories: http://bugs.darcs.net/issue1938 . We will try to address these issues by the release of 2.8. We are very interested in knowing whether there are other issues, so please let us know on the bug tracker or by commenting this post.

    Okay, we talked about a "un-feature", but Darcs 2.8 will also contain a few new features that will probably please a lot of you. This will be for another blog post very soon.

    Posted by at 1 comment:

    Monday, November 22, 2010

    darcs weekly news #79

    News and discussions

    1. Eric called for a Windows darcs packager:
    2. The lists darcs-users and darcs-devel now follow a more usual separation, that is, patches discussions are now being done on darcs-devel. This will make darcs-users less noisy for non-developers:
    3. Gour explained why he switched from darcs to Fossil:

    Issues resolved in the last week (6)

    issue332 Gabriel Kerneis
    issue1397 Alexey Levan
    issue1637 Dmitry Tsygankov
    issue1965 Reinier Lamers
    issue1970 Florent Becker
    issue1988 Gabriel Kerneis

    Patches applied in the last week (89)

    See darcs wiki entry for details.
    Posted by at 1 comment:
    Labels:

    Sunday, November 14, 2010

    darcs weekly news #78

    News and discussions

    1. Reinier talked about the release process of darcs 2.4 and 2.5:
    2. Petr released darcs-fastconvert 0.2:

    Issues resolved in the last week (2)

    issue1551 Eric Kow
    issue1977 Guillaume Hoffmann

    Patches applied in the last week (25)

    See darcs wiki entry for details.
    Posted by at 2 comments:
    Labels:

    Sunday, November 7, 2010

    darcs weekly news #77

    News and discussions

    1. Simon Michael worked on adding support for darcs repositories for ohloh.net and called for volunteers:
    2. Christian Maeder asked how to apply a darcs patch to a non-darcs source tree:

    Issues resolved in the last week (2)

    issue1266 Alexey Levan
    issue1984 Dmitry Tsygankov

    Patches applied in the last week (14)

    See darcs wiki entry for details.
    Posted by at 2 comments:
    Labels:

    Monday, November 1, 2010

    darcs weekly news #76

    News and discussions

    1. Darcs 2.5 was released!

    2. The report from the Orleans Sprint was posted:

    Issues resolved in the last week (3)

    issue182 Eric Kow
    issue734 Gabriel Kerneis
    issue1809 Ganesh Sittampalam

    Patches applied in the last week (51)

    See darcs wiki entry for details.
    Posted by at 1 comment:
    Labels:

    Thursday, October 28, 2010

    darcs hacking sprint 5 report

    The fifth Darcs hacking sprint took place in Orleans over the weekend of 15-17 October.

    We seem to be starting a tradition of sprints coinciding with social movements. Last year, the sprint venue in Vienna was squatted by students protesting university fee reforms. This time we were caught in the French pension reform strikes, which knocked out one of our would-be participants and made another lose a day.

    The sprint was small but productive. We had four people attending, Florent Becker (also local organizer), Guillaume Hoffmann, Eric Kow and Reinier Lamers.

    Talks and discussions

    Scribble Scribble Think Think!

    Maintaining Darcs

    The day before the sprint, Eric gave a talk to undergraduate and masters students on Free and Open Source Software Projects, in particular the principles that we try to apply within the Darcs team.

    Therapy session

    We kicked off the sprint with a discussion of some of the challenges that we have been facing in the Darcs community and how we hope to rise to them over the long term.

    1. Code quality - darcs code is a gem buried in big pile of muck. We've been making progress tidying the mess and moving towards a clean, well thought out library... but we still have a long way to go.

    2. Portability - darcs relies on GHC, which takes a long time to build and which simply does not support certain niche platforms.

    3. Usability - for all its power, Darcs has a reputation among its fans for being exceptionally easy to use. While we can be proud of our friendly UI and simple mental model, we need to also recognise the parts of Darcs that make life difficult. Here are three areas we should explore:

      • Patch annotations would allow Darcs to start tracking a repository history while still allowing for patch reordering. People should be able to ask questions like "who signed off on this patch?", and "when was in pulled into the current repository?"

      • Short version identifiers would make it easier for users to communicate with each other - "Bob, could you please fetch version 83dc9fa3?"

      • Better conflict marking and summaries would help maintainers to merge sets of patches from long term branches.

    4. Network effects - Darcs is most useful when many other people are also using Darcs or something compatible.

      • Unlike Git/Hg/Bzr, Darcs lacks bridges with the other DVCSes. The other three can more or less talk to each other because they have similar models. Darcs is a bit different, so making good bridges can be tricky.

      • The Darcs community lacks services that facilitate collaboration to the same extent that Github does. We need patch-tag, darcsden and friends to get better still!

      • One piece of low-hanging fruit to pluck is the ability to host Darcs repositories on servers that lack a Darcs binary. How do we push patches over SFTP without the luxury of a remote Darcs binary?

    File formats

    We continued the discussion from the darcs-users mailing list on machine-readable formats for the Darcs data and command output. This discussion was tricky because it involves many different parts of Darcs and involves juggling some conflicting goals:

    • Standards compliance: preference for well-documented and understood standards.
    • Familiarity: attention to de-facto standards used in the revision control community.
    • Conceptual integrity: all Darcs formats should work the same way
    • Extensibility: ability accomodate future requirements
    • Easy parsing: people should be able to whip up quick little scripts to slice and dice Darcs output
    • Agnosticism: need a good story for arbitrary bytes because Darcs is fairly agnostic to the content of text files
    • Transparency: no complicated escaping mechanisms as these tend to fall in rare corner cases and would be easy to get wrong.
    • Conservatism: the less we change Darcs, the better

    Reinier had the idea of bringing this discussion to a whiteboard -- this is why you need hackathons! -- which allowed us to take a more global view of the problem. After much discussion, we reached a conclusion was that we should converge on 4 formats:

    1. Unified diff format [SAME], (familiarity, conservatism) there's no reason to move away from diff/patch style output for low-level diffs.

    2. High level patch format [SAME] (conceptual integrity, agnosticism, conservatism) - this is a high-level representation of patches which is unique to Darcs. For instance, it can describe file renames and word replacement. We plan to continue using this format whenever we need to represent high-level patch contents.

    3. Line-separated annotate format [NEW] (easy parsing, agnosticism, transparency) - We will deprecate the annotate --xml format, and shift to a line-based one. If there are community standards that exist we'll try to use them as much as possible.

    4. Hashed context file format [NEW] (agnosticism, transparency, conceptual integrity) - we will deprecate the changes --xml format and converge to an extended version of the context file format. New features:

      • file contents hashes (issue1550)
      • format version information
      • is-context-file flag (need deps to be safe to use)

    Note that where forced to choose, we have essentially sacrificed the otherwise worthy goals of standards compliance and extensibility.

    Hacking

    Reinier and Guillaume hacking away

    Darcs 2.5

    Darcs 2.5 is almost here! The release was delayed for quality control reasons, but after many betas and bug fixes, we think we're ready to ship. Reinier put the finishing touches on our first release candidate.

    Infrastructure

    Eric made a handful of improvements to the issue tracking infrastructure, improving integration with our darcs repository and darcswatch.

    User interface

    Eric and Reinier polished off some user interface work:

    • Removed a confirmation prompt asking you if you really want to record your patch when you choose to edit long comment but make no changes. (undo beats confirmation).
    • Testing UI regression fix by Adolfo: Darcs was overzealous in warning of about unreachable cache entries.
    • Improved checking of commands that work on file paths

    Pristine cache handling

    Guillaume documented much of Darcs pristine cache handling, fixing a darcs repair bug along the way. He

    1. Studied the problem of garbage collecting the Darcs pristine cache http://wiki.darcs.net/Using/GrowingPristineProblem
    2. Fixed darcs repair when pristine cache is missing.
    3. Designed some improvements to darcs get handling of missing pristine cache items. http://bugs.darcs.net/issue1976

    No working directory: towards passive repositories

    We want to make it as easy for people to use and host Darcs repositories. In particular, we think it would be great if you could host a Darcs repository on any server, without caring if Darcs is installed there or not. While it is already possible to fetch Darcs repositories from such server, what we now need is the ability to push to such repositories without a remote copy of Darcs.

    Working in this direction, Florent implemented a long-requested feature for repositories without a working directory. This is useful for repositories which are only meant to be used for pushing/pulling, where the notion of a working directory is superfluous and makes some Darcs operations harder to implement.

    Faster annotate (we'll get there!)

    Unfortunately, Benedikt could not join us for the sprint as travel from Zurich to Orleans was disrupted by strikes. Luckily, he was still able to participate over IRC. He ported over his work on the "patch index" optimisation to the latest version of the Darcs code in progress (that's a 6 month leap!) and will continue by exposing the patch index to Darcs commands.

    Experience report

    Guillaume (right) with a question for Reinier

    Our Darcs Weekly News editor attended his first sprint 6 months ago in Zurich, starting work on some ProbablyEasy bugs. It was great to see him again and very encouraging to see how much deeper he was getting into Darcs internals. Let's hear it from Guillaume:

    I arrived at the sprint with this bug report in mind, written by a NetBSD user who could not build Darcs on his system. I wondered how easy could it be to write a minimal Darcs client that could only fetch a working copy from a Darcs repository, in a programming language more common than Haskell (Python comes to mind).

    Thus began my discovery of the hashed repository format. The most susprising thing I discovered was the lack of documentation: currently someone who wants to write a Darcs client can only count on the existing source code. So I started to document what I understood by asking the other sprinters and looking at the code.

    I also documented the Growing Pristine Problem as it was cited as being a low point of the hashed repository format with regards to the old-fashioned format. After understanding why this phenomenon happens, I believe that this is an unavoidable issue when one wants to avoid breakage during simultaneous pushing and getting the same repository. Also, it becomes a problem only in really big repositories.

    However, some parts of Darcs could be improved. Darcs could do a better work to handle its pristine.hashed files. For instance, as of now, deleting the pristine.hashed directory leads to an almost dead-end situation since "darcs repair" refuses to work unless a dummy pristine.hashed directory is created. I sent a test case and a fix for this problem.

    Missing pristine files are generally not handled graciously by Darcs while their presence is not necessary (albeit very important for speed). As of now, "darcs get" refuses to work when a pristine file is missing, and this has already bit me in the past. I proposed an enhancement of this behaviour. Other local commands that use the pristine files fail if one file is missing, but never tell the user to run darcs repair. I will probably work on these two proposals soon.

    The aim is to make Darcs as robust as possible with its current format, and above all to prevent users from being exposed to unhelpful error messages.

    Thanks

    Thanks to Florent and to the rest of the laboratoire LIFO for hosting the Darcs team this weekend! Hosting sprints is an excellent way to support and to interact with the Darcs community.

    The obligatory Jeanne D'arc statue photo

    A special shout-out also goes to Yannick Parmentier, a LIFO researcher (and coincidentally Eric's former office mate) who very kindly visited us to take photos and shuttle us back and forth between Orleans and the lab. Merci, Yannick!

    Merci, Yannick!

    See you next time!

    This was a really fun sprint. We hope you can join us next time, hopefully in 6 or so months. In the meantime, check out the flickr tag darcs-2010-10 for more photos from the sprint.
    Posted by at 12 comments:
    Labels:

    Sunday, October 24, 2010

    darcs weekly news #75

    News and discussions

    1. Reiner releases the release candidate 1 and the packager's preview 1 of Darcs 2.5:

    2. Florent and Ganesh volunteered to be the next co-release managers:

    3. Finally, the Darcs Hacking Sprint took place last week in Orleans. We will soon post a report.

    Issues resolved in the last week (3)

    issue1942 Dmitry Astapov
    issue1948 Eric Kow
    issue1951 Reinier Lamers

    Patches applied in the last week (22)

    See darcs wiki entry for details.
    Posted by at 1 comment:
    Labels:

    Monday, October 4, 2010

    darcs weekly news #74

    News and discussions

    1. Reinier updated us about the state of darcs 2.5:
    2. Eric proposed a way to reduce non-user related traffic on the darcs-users mailing list:
    3. Ganesh tried to build darcs 2.5 with GHC 7.0:

    Issues resolved in the last week (0)

    Patches applied in the last week (3)

    See darcs wiki entry for details.
    Posted by at 2 comments:
    Labels:

    Sunday, September 19, 2010

    darcs weekly news #73

    News and discussions

    1. Ganesh proposed a simple change in conflict marking:

    2. The commiters of the Darcs team now use an additional public branch for sharing unreviewed or lightly reviewed work in progress:

    Issues resolved in the last week (0)

    Patches applied in the last week (4)

    See darcs wiki entry for details.
    Posted by at 1 comment:
    Labels:

    Monday, September 13, 2010

    darcs weekly news #72

    News and discussions

    1. Darcs 2.5 beta 5 was released:

    2. Eric put online a series of videos (~1h) explaining the sate of darcs in 2010 (adapted from his recent talk in AngloHaskell 2010):

    3. Petr released darcs-fastconvert, a tool that helps converting darcs and git repositories in both directions:

    4. Jason asked for feedback about his proposition for darcs to uniquely identify the state of a repository by a hash:

    5. Petr kicked off his ``adventure'' branch, a long-lived public branch of darcs he will use to make disruptive changes in the codebase:

    6. Ganesh proposed to used a ``submitted'' public branch to help developers share they work and deal with the reviewing process more smoothly:

    Issues resolved in the last 3 weeks (2)

    issue1884 Alexey Levan
    issue1942 Petr Rockai

    Patches applied in the last 3 weeks (140)

    See darcs wiki entry for details.
    Posted by at 1 comment:
    Labels:

    Monday, August 23, 2010

    darcs weekly news #71

    News and discussions

    1. Darcs 2.5 beta 4 was released:

    2. Ganesh put online his branch of darcs containing the ``rebase'' command for all to try. Simon Marlow gave feedback on this feature:

    3. Joachim Breiner uploaded ipatch on hackage:

    Issues resolved in the last week (3)

    issue1875 Eric Kow
    issue1898 Eric Kow
    issue1913 Ganesh Sittampalam

    Patches applied in the last week (31)

    See darcs wiki entry for details.
    Posted by at 1 comment:
    Labels:

    Sunday, August 15, 2010

    darcs weekly news #70

    News and discussions

    1. Darcs 2.5 beta 3 was released:

    2. The next Darcs Hacking Sprint will take place at Orleans, France, the 15-17 of October:

    3. Version 0.1.9 of darcs-benchmark was released, including bugfixes and experimental support for comparison with git and mercurial:

    4. Adolfo Builes blogged about the end of his summer of code project. A summary of his project is available on the wiki:

    Issues resolved in the last week (7)

    issue1290 Eric Kow
    issue1530 Eric Kow
    issue1599 Adolfo Builes
    issue1873 Petr Rockai
    issue1896 Ganesh Sittampalam
    issue1908 Petr Rockai
    issue1909 Petr Rockai

    Patches applied in the last week (49)

    See darcs wiki entry for details.
    Posted by at 1 comment:
    Labels:

    Friday, August 6, 2010

    darcs weekly news #69

    News and discussions

    1. darcs 2.5 beta 2 was released. Give it a try!

    2. Reinier explained why beta 3 will have to wait a little:

    3. Joachim "nomeata" Breitner released a ipatch, a tool based on darcs' hunk editing feature:

    4. Two more blog posts from our Summer of Code students Adolfo and Alexey:

    Issues resolved in the last week (2)

    issue1888 Petr Rockai
    issue1892 Petr Rockai

    Patches applied in the last week (38)

    See darcs wiki entry for details.
    Posted by at 1 comment:
    Labels:

    Saturday, July 24, 2010

    darcs weekly news #68

    News and discussions

    1. ``darcs stash'' was discussed, with different possible implementations and UI proposed, and some example workflows evoked:

    2. Reinier listed the release blockers for darcs 2.5:

    Issues resolved in the last week (4)

    issue1716 Reinier Lamers
    issue1883 Eric Kow
    issue1887 Petr Rockai
    issue1893 Ganesh Sittampalam

    Patches applied in the last week (42)

    See darcs wiki entry for details.
    Posted by at 1 comment:
    Labels:

    Sunday, July 11, 2010

    darcs weekly news #67

    Issues resolved in the last week (6)

    issue1288 Ganesh Sittampalam
    issue1726 David Markvica
    issue1825 Petr Rockai
    issue1845 Petr Rockai
    issue1865 Petr Rockai
    issue1871 Petr Rockai

    Patches applied in the last week (83)

    See darcs wiki entry for details.
    Posted by at 1 comment:
    Labels:

    Monday, June 28, 2010

    darcs weekly news #66

    News and discussions

    1. Reinier announced the release schedule of Darcs 2.5 (soft freeze July 8th, release August 7th):

    2. Reiner also issued a call for volunteers for fixing unassigned bugs that should be fixed for the next release:

    3. Eric explained how to make the patch reviewing process more efficient:

    4. And the Sumer of Code blog posts of the last two weeks:

    Issues resolved in the last week (7)

    issue1176 Adolfo Builes
    issue1277 Eric Kow
    issue1389 Reinier Lamers
    issue1713 Eric Kow
    issue1857 Petr Rockai
    issue1877 Florent Becker
    issue1879 Eric Kow

    Patches applied in the last week (53)

    See darcs wiki entry for details.
    Posted by at 4 comments:
    Labels:

    Saturday, June 12, 2010

    darcs weekly news #65

    News and discussions

    1. Lele Gaifax released a new version of the trac+darcs plugin:

    2. We are still looking for a release manager for the release of darcs 2.5. Eric summarized the discussions concerning recruitment and the responsibilities and challenges related to this job:

    3. Roadmap: rebase won't be in 2.5, annotate will be improved:

    Issues resolved in the last week (2)

    issue1210 Adolfo Builes
    issue1874 Eric Kow

    Patches applied in the last week (21)

    See darcs wiki entry for details.
    Posted by at 1 comment:
    Labels:

    Sunday, June 6, 2010

    darcs weekly news #64

    News and discussions

    1. An alpha release of darcs 2.5 might happen soon:

    2. Summer of Code: Alexey Levan sent a first version of patches for darcs optimize --http, and Adolfo Builes sent a patch fixing a bug concerning cache pool choice by darcs:

    Issues resolved in the last week (10)

    issue1337 Petr Rockai
    issue1503 Adolfo Builes
    issue1610 Petr Rockai
    issue1817 Petr Rockai
    issue1839 Florent Becker
    issue1843 Florent Becker
    issue1848 Florent Becker
    issue1860 Petr Rockai
    issue1861 Eric Kow
    issue1864 Florent Becker

    Patches applied in the last week (18)

    See darcs wiki entry for details.
    Posted by at 1 comment:
    Labels:

    Sunday, May 30, 2010

    darcs weekly news #63

    News and discussions

    1. Darcs 2.4.4 was released this week:

    2. Eric talked about the ongoing work in fixing the bug about non-ASCII filenames:

    3. Adolfo wrote his first Google Summer of Code blog report:

    Issues resolved in the last week (2)

    issue1763 Petr Rockai
    issue1784 Guillaume Hoffmann

    Patches applied in the last week (14)

    See darcs wiki entry for details.
    Posted by at 1 comment:
    Labels:

    Tuesday, May 18, 2010

    darcs weekly news #62

    News and discussions

    1. Unexpectedly, we got a second student funded by the Google Summer of Code this year. Adolfo Builes will work on improving the global cache:

    2. Darcs 2.4.4 is going to be released in one week if no bug is discovered in its current source (most importantly under Windows):

    3. We are looking for a new release manager:

    Issues resolved in the last week (1)

    issue1841 Jeremy Cowgar

    Patches applied in the last week (22)

    See darcs wiki entry for details.
    Posted by at 1 comment:
    Labels:

    Sunday, May 9, 2010

    darcs weekly news #61

    News and discussions

    1. Eric announced the release of darcs 2.4.3, which fixes critical bugs under Windows and fixes the performance regression of darcs convert:

    2. Simon Michael proposed a cleanup of the repository format names and gathered a few answers and proposals:

    1. Ganesh and Petr explained why it is not recommended anymore to build darcs with GHC 6.8:

    Issues resolved in the last week (7)

    issue64 Reinier Lamers
    issue1232 Dino Morelli
    issue1760 Petr Rockai
    issue1769 Matthias Kilian
    issue1814 Petr Rockai
    issue1824 Eric Kow
    issue1837 Petr Rockai

    Patches applied in the last week (115)

    See darcs wiki entry for details.
    Posted by at 1 comment:
    Labels:

    Saturday, April 24, 2010

    darcs weekly news #60

    News and discussions

    1. darcs 2.4.1 was released, fixing a couple of bugs of version 2.4.0. However, a serious bug under Windows was discovered, so Windows users should still stick to the 2.3.1 version, while 2.4.2 is not out:

    2. Three students wrote their applications for this year's Summer of Code. Projects members discussed the priorities of darcs: network speed, local speed or UI ?

    Issues resolved in the last week (3)

    issue1159 Dmitry Kurochkin
    issue1645 Eric Kow
    issue1756 Reinier Lamers

    Patches applied in the last week (29)

    See darcs wiki entry for details.
    Posted by at 1 comment:
    Labels:

    Sunday, April 4, 2010

    darcs weekly news #59

    News and discussions

    1. Lennart Kolmodin announced that darcs 2.4 was available in Gentoo Linux:

    2. Mark Stosberg talked about darcs handling large repositories, then spawning a discussion on git vs darcs from the UI point of view:

    3. If you haven't already read it on this blog, you can have a look at Eric's report from the last darcs hacking sprint:

    Issues resolved in the last week (4)

    issue1208 dixiecko@gmail.com
    issue1427 Guillaume Hoffmann
    issue1739 Reinier Lamers
    issue1756 Petr Rockai
    issue1765 Reinier Lamers

    Patches applied in the last week (29)

    See darcs wiki entry for details.
    Posted by at 2 comments:
    Labels:

    Sunday, March 28, 2010

    darcs hacking sprint 4 report

    Updated 2010-03-29 with more photos (thanks, David Anderson!), a small correction and a note about the SFC

    The Fourth Darcs Hacking Sprint took place last weekend (19 to 21 March) as part of the Zurich Haskell Hackathon. We had a very productive sprint, a bit of code written, polished off many key discussions, had a little beer and a lot of fun.

    Overview

    In this sprint, we worked on finishing some performance work for the upcoming Darcs 2.5 release this summer (hashed storage, patch index, global caches, inventory hashing); planning our work for the Darcs 2.6 release next year (smart servers, cache cleanup, darcs rebase) and working with new users of the Darcs library.

    Issues resolved

    • issue643 darcs send -o output - Guillaume Hoffmann
    • issue1473 annotate command line - Stefan Wehr
    • issue1456 portable darcs dist - Guillaume Hoffmann

    New Darcs Hackers

    We're always happy to work with new Darcs developers. At this sprint, we were joined by four new contributors.

    Guillaume Hoffmann

    Guillaume has been writing our Darcs Weekly News articles for a year now. Over the weekend he got his first taste of Darcs hacking, knocking out three ProbablyEasy bugs (darcs dist internals, darcs send -o UI, darcs apply with gzipped patch bundles). Guillaume reports that he can see himself doing more of this in the future!

    Steven Keuchel

    Steven worked on a new feature to display the file contents hashed associated with any patch. This makes it easier for third party tools to inspect the patch files behind Darcs.

    Stefan Wehr and David Leuschner

    Stefan and David mostly worked on the Darcs Patch Manager, but to warm up, they tackled a couple of ProbablyEasy bugs, particularly a bug in darcs annotate that was affecting Redmine

    Hacking continued...

    Darcs hackers at work (Saturday)
    Photo taken from David Anderson's Picasa site

    Bugfix: Darcs on Windows shares

    Salvatore tracked down the Windows regression on 2.4 that make Darcs not work on windows shares.

    Performance: Fast darcs annotate

    Benedikt Schmidt continued his work on the patch index (formerly known as the filecache). The patch index keeps track of which patches affect which files. This index will bring a big boost to darcs annotate performance, particularly for files which are affected by relative small number of patches.

    Performance: Global cache

    Luca continued his work on breaking up the global cache ($HOME/.darcs/cache) into buckets for faster access. Working with Reinier and Petr, Luca has developed an approach to migrating from old style caches to the new style bucketed ones. He has also improved the implementation to use hard links, to avoid disk space doubling and to preserve backwards compatibility with prior versions of Darcs.

    Windows installer

    Salvatore put together a nice Windows installer using the bamse package. It looks like we will be able to use this for the planned Darcs 2.5 release this summer. This work will also open the door to nicer integration with Windows tools, for example, using a bundled Tortoise SSH for better experience working with SSH passphrases.

    Interactive cherry picking

    Florent improved the quality of the Darcs cherry picking code, making it easier to fine tune our user interface and some day support graphical interfaces via the Darcs library. Witnessed list zippers for the win?

    Interactive diff

    Florent also started work on adding Darcs's interactive cherry picking to darcs diff, making it possible to choose a set of patches to view as a diff.

    Performance: Hashed storage completion

    Darcs has a representation of file and directory trees called slurpies. Petr polished off his work to replace the slurpies with his more efficient, general purpose hashed-storage library. Slurpies are going away, and Darcs will be faster for it. He and Ganesh also discussed how to gracefully transition from repositories created before the hashed-storage refactor.

    Performance: Using tags when writing patches

    Petr ported work by David Roundy to solve a scalability regression in hashed repositories. For darcs commands that write out patches, we had a naive hashing operation that does not account for the fact that patches behind tags cannot be modified. Darcs was unnecessarily traversing the entire sequence of patches (ie. O(n) time) when it could easily have been just traversing the sequence since the last tag.

    UTF-8 metadata

    Reinier continued to improve the encoding of Darcs patch metadata. Darcs is completely agnonstic with respect to the encoding of your files. Unfortunately, this agnostism extends to patch metadata (patch name, patch author), making it difficult for people to collaborate across different locales. To address this problem, Reinier has been working to make Darcs store its patch metadata in a single encoding (UTF-8) while gracefully supporting older patches (with metadata in potentially any encoding).

    Discussions

    The rebase discussion
    Also from David's site

    Release process

    The Darcs 2.4 release was quite a tricky one to navigate. We found that bugs were only being flushed out on release candidate time and sometimes after the release proper.

    We would like to encourage more people to try out Darcs work in progress and give us feedback early in the release process. After chatting about this, Reinier (with Ganesh, Eric and Petr) decided that as Release Manager, he would put out a Darcs alpha every 4 weeks.

    In the future we may investigate automatic nightly builds via the buildbot and a platform support policy such as the one used by Tahoe.

    Darcs patch index (fast darcs annotate)

    Benedikt updated us on the recent status of his ongoing patch index work (formerly known as the filecache). We discussed the things that make the patch index convincing (permanant, repo-local, unique identifiers for files) the interaction between the patch index and the type witnesses and also ways of tuning the patch index performance and keeping it small.

    We're looking forward to sharing the new patch index optimisation with you in upcoming releases. Darcs annotate may become a lot more useful in the next couple of releases!

    Readable darcs annotate

    Fast darcs annotate won't be useful if nobody can read it. Benedikt and Eric worked on designing a better output format darcs annotate. Taking a page from git blame, there will be one line per source file line, with columns for patch identifier, author name, date and finally the line. One of the design questions was how we should best refer to darcs patches, the current best candidate being a prefix of the darcs patch metadata hash.

    Fast darcs over networks

    Darcs get over networks is slow, painfully slow. Petr has suggested two priorities for improving the performance of network operations. The first would be to introduce a darcs optimize --http feature which would optimise the Darcs repository for fetching over a network (for example, by creating a "snapshot" of the pristine cache to be fetched in one go). The second priority would be develop a smart server that would provide darcs clients with only the files they need and in the optimal number of chunks. The two ideas combined would make an excellent Google Summer of Code project.

    Darcs rebase

    Prior to the sprint, Ganesh was working on a darcs rebase feature. Rebase will help Darcs users work with long term branches, and other cases where patch commutation by itself is not enough. At the sprint, Ganesh explained his work to everyone interested. Together we settled on a rough plan for the user interface. It looks like our new rebase command will offer a typically Darcs-ish twist: interactive cherry picking.

    Darcs library

    Ganesh and Florent talked with three teams building software in the Darcs ecosystem (DPM: Stephan Wehr and David Leuschner, Mac Darcs record GUI: Benedikt Huber and David Markvica, DarcsDen: Alex Suraci). There was a surprising degree of commonality.

    The conversations have given us a much stronger sense of direction with the Darcs library. In particular, Ganesh is convinced that we should commit to our use witnesses - at the very least getting them completely finished so we can run with them, probably turning them on by default, and quite possibly dropping the non-witnesses builds.

    Default switches

    We held a quick roundtable discussion to settle some decisions on Darcs default switches that have been hanging in the air. Our decisions for Darcs 2.5:

    • --no-set-scripts-executable [unchanged]
    • pull/push/send --no-set-default
    • send --edit-description
    • record --no-test
    • check --no-test

    Performance presentations

    Petr and Benedkit gave lighting talks, showing some of our recent performance work to the Haskell community. Some exciting numbers from Benedikt's work (notes) include a 6 second darcs annotate on a file in the GHC repository (previously this did not complete within a half hour).

    Google Summer of Code

    We discussed our priorities for this year's Google Summer of Code. We have decided that we would focus our attention on performance issues. If we had two GSoC students this year, we would be mainly interested in dividing them between

    network performance

    developing a smart server for much faster darcs get and pull over a network

    local performance

    performing a comprehensive overhaul of the Darcs hashed file cache handling

    We also discussed ways to make the best use of our students' time. The Darcs team has participated in GSoC twice and learning a lot from the experience. This year we would like to see if we could publish some clear guidelines both on what we expect from GSoC students and what they can expect from us. Watch the mailing list for more discussion on this topic.

    Budding Ecosystem

    We were pleasantly suprised to find ourselves with users of the (still unstable) Darcs API. These new arrivals give us the feeling that the collection of related software is coalescing into a new Darcs ecosystem.

    Darcs Patch Manager

    David Leuschner and Stefan Wehr worked on an exciting new patch management program for project maintainers. The Darcs Patch Manager (DPM) offers a new way for repository maintainers to keep track of incoming Darcs patches, including their amendements and dependencies.

    $ dpm -r MAIN_REPO -s DPM_DB list
    very cool feature [State: OPEN]
    2481 Tue Mar 16 17:50:23  2010 Dave Devloper <dave@example.com>
    State: UNDECIDED, Reviewed: no
    added
    7861 Tue Mar 16 17:20:45  2010 Dave Devloper <dave@example.com>
    State: REJECTED, Reviewed: yes
    marked as rejected: one minor bug
    some other patch [State: OPEN]
    7631 Tue Mar 16 13:15:20  2010 Eric E. <eric@example.com>
    State: REJECTED, Reviewed: yes
    added
    ...

    Towards the end of the hackathon, Stefan gave a nice short demo of DPM in action and deftly avoided the wrath of the demo Gods.

    MacOS X GUI for Darcs record

    Benedikt Huber and David Markvica started work on a graphical interface to the Darcs record command. One key twist is that they make use of the Darcs API to get the kind of dependency-tracking interactiveness goodness that Darcs offers. Bendedikt and Huber report that they have spent most of the hackathon getting to grips with the library. Darcs type witnesses were very helpful for avoiding errors, but they also impose a steep learning curve.

    Darcsden

    Alex Suraci and Simon Michael made several improvements to Darcsden, an open source hosting solution (akin to Github and Patch-tag). Some recent changes were Atom feeds, the ability to view forks of your repository and cherry-pick patches from them (work in progress). Darcsden also makes use of the Darcs API.

    Darcsden fork viewer

    Want to host Darcs Hacking Sprint 2010-10?

    The Darcs Team would like to hold hacking sprints twice a year. These sprints are an important occassion for us to hold design discussions, hack some code, train new Darcs hackers and generally bond as a team.

    Do you think you can help? Please get in touch with me if you think you may be able to host a group of around 20 Darcs hackers one of these October or November weekends.

    Thanks!

    Getting over 75 Haskell hackers into Zürich and having them up and running on arrival (Swiss power plugs notwithstanding) was no easy task! We'd like to thank Johan Tibell, David Anderson and the rest of the Google Crew for their hard work organising this hackathon.

    Thanks also to the generous donors who chipped into our 2010 Darcs Travel Fund. We'll be looking forward to using the leftover cash for the upcoming 5th Darcs Hacking Sprint in October or November.

    Speaking of donors, we'd particularly like to thank the Software Freedom Conservancy for providing us with the infrastructure (both legal and technical) for accepting donations and holding assets such as the darcs.net domain. Meta projects like the SFC are crucial for the success of volunteer-driven open source projects such as Darcs.

    Finally here are some words from happy Darcs hackers:

    The sprint was a wonderful social occasion, and it was great meeting most of the Darcs hackers, and also seeing other Haskell hackers interested in working in the Darcs ecosystem. I especially enjoyed teaching them how to use our API. -- Florent

    The atmosphere was wonderful and I consider the sprint to have been very productive overall. -- Petr

    This is coolest thing I ever did -- Luca

    See you in half a year!

    Participants

    We had ten Darcs hackers in Zürich along with four Haskellers using the Darcs API to do awesome things (plus two more on IRC).


    • Florent Becker
    • Guillaume Hoffmann
    • Eric Kow
    • Reinier Lamers
    • Ganesh Sittampalam
    • Petr Rockai
    • Salvatore Insalaco
    • Luca Molteni
    • Benedikt Schmidt
    • Steve Keuchel
    • Benedikt Huber
    • David Markvica
    • Stefan Wehr
    • David Leuschner
    • Simon Michael [IRC] - Darcsden
    • Alex Suraci [IRC] - Darcsden
    (Group photo from Johan Tibel's ZuriHac summary post)


    Posted by at 20 comments:
    Labels:
    Subscribe to: Posts (Atom)

    Blog Archive

    Followers