--blob-exec. Improved code originally developed by Paul Draper #213
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
It was not properly waiting for the child processes, making it to hang under some architectures. At this point I am hardcoding a lot of the processing (e.g. process only .[ch]^ files
it now takes two parameters: - the command name - a mask It also checks that the command name exists as a file and if no mask is given it will abort
|
Could you give some examples how to use it? What is the key and what should be the value? |
|
Wilson Young <[email protected]> writes:
Could you give some examples how to use it?
--blob-exec:<key>=<value>
execute the system command for each
blob
What is the key and what should be the value?
i didn't know how to add two different parameters to the command
line, so I hacked it.
…--blob-exec:pathToScript=<regularExpressionOfFilesToProcess>
The first is the script to run for each file blob. The second is
the regular expression to determine if the script is run on the specific blob. The script is
run if the regular expression matches its filename (no path included).
Note that this script reads from stdin and outputs to stdout. It
is expected to read other information from two environment variables:
# BFG_BLOB: the id of the blob (a SHA)
# BFG_FILENAME: basename of the file to process. it has an
extension
remember that the current implementation of BFG does not keep the
full path of the file.
hopefully this will make it easier to use it. Check github/cregit
for use of this.
There might be a bug in my patch that only allows the execution of
bfg from the main directory of the repository, but I have not debugged it.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the
thread.
--
Daniel M. German "It is good to be alive"
Malcolm
http://turingmachine.org/
http://silvernegative.com/
dmg (at) uvic (dot) ca
replace (at) with @ and (dot) with .
|
|
@dmgerman I tried your RP with conflicts resolved with latest master. |
|
Wilson Young <[email protected]> writes:
@dmgerman I tried your RP with conflicts resolved with latest
master.
It getting compiled, and all other functions works, but the
added blobexec cannot getting though:
bfg --blob-exec:clean=js$ cloud-service-connector.git
command clean mask js$
Error: Option --blob-exec failed when given
'clean=js$'. None.get
in the command line? you might need to escape the $ . Try
bfg --blob-exec:clean='js$' cloud-service-connector.git
and let me know if that works. $
|
|
@dmgerman No difference, I tried both without $ or quoted. Always |
|
I just tried. I think the bug has to do with running the command outside
the root git directory:
dmg@iodine:/tmp|⇒ java -jar
/home/dmg/git.hacking.mine/bfg-repo-cleaner/jars/bfg.jar
'--blob-exec:/usr/bin/head=\.c$' xournal
command /usr/bin/head mask \.c$
Error: Option --blob-exec failed when given '/usr/bin/head=\.c$'. None.get
but if you run it in the root directory it would work:
dmg@iodine:/tmp/xournal|master ⇒ java -jar
/home/dmg/git.hacking.mine/bfg-repo-cleaner/jars/bfg.jar
'--blob-exec:/usr/bin/head=\.c$' .
command /usr/bin/head mask \.c$
command /usr/bin/head mask \.c$
Using repo : /tmp/xournal/./.git
Found 125 objects to protect
Found 27 commit-pointing refs : HEAD, refs/heads/master,
refs/remotes/origin/HEAD, ...
Protected commits
-----------------
|
|
@dmgerman The command works on git dir, but got another error: Any clue? |
|
Wilson Young <[email protected]> writes:
@dmgerman The command works on git dir, but got another error:
```
Cleaning commits: 59% (161/272)Exception in thread
"Thread-1686" java.io.IOException: Broken pipe
at java.base/java.io.FileOutputStream.writeBytes(Native
Method)
at
java.base/java.io.FileOutputStream.write(FileOutputStream.java:355)
at
java.base/java.io.BufferedOutputStream.write(BufferedOutputStream.java:123)
at
java.base/java.io.FilterOutputStream.write(FilterOutputStream.java:108)
at
com.madgag.git.bfg.cleaner.BlobExecModifier.writeJob$1(BlobExecModifier.scala:44)
at
com.madgag.git.bfg.cleaner.BlobExecModifier.$anonfun$execute$4(BlobExecModifier.scala:49)
at
com.madgag.git.bfg.cleaner.BlobExecModifier.$anonfun$execute$4$adapted(BlobExecModifier.scala:49)
at
scala.sys.process.ProcessBuilderImpl$Simple.$anonfun$run$2(ProcessBuilderImpl.scala:75)
at
scala.sys.process.ProcessImpl$Spawn$$anon$1.run(ProcessImpl.scala:23)
```
Any clue?
I might have seen this before... it might be that for certain
blobs your script is failing.
I would first try it with a very narrow regular expression
(perhaps matching one file) and
see if it succeeds.
…---dmg
|
|
Note: blobexec command is nice, however it might be complicated to make the command reproducible across various operating systems. For instance. Suppose one runs I have implemented "blobexec" + store the result of processing, so the results could be reproduced even without running the command. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
hi Roberto,
I have improved the code by Paul Draper. The main issue is that his original code did not properly wait for the children to complete. This lead to a crash under linux (see #169).
I also added the ability to specify a regular expression to indicate the files that should be processed (my own bias, since I don't like file masks, as they are not
as powerful as regexps). I have been using this patch to process several large repos (including linux and git) and it seems to be working well.
The script to execute should read the file from stdin, and output to stdout. Two environment variables are used to communicate between BFG and the script: BFG_BLOB (contains the blobid of the file to process) and BFG_FILENAME (contains the basename, including extension) of the file to process.
If the following conditions happen, the blob in the commit is not replaced:
This pull request is a superset of #83 and addresses #169.
Regarding licensing: my commits are licensed under the same license as BFG. Paul Draper submitted his commits in another pull request, who I presume, licensed the patch according to the contributing guidelines (https://github.com/rtyley/bfg-repo-cleaner/blob/master/CONTRIBUTING.md)
--dmg