force language detection for files in .gitignore

[Icyrockton]

I'm implementing a crawler.
I want to use linguistic command line single-file mode to detect the language of certain downloaded files, but the directory of these downloaded files are annotated in .gitignore (e.g., download/ ), and linguistic will not be able to find these files and report an error, but locally I own these files, can I force a bypass of the .gitignore restriction?

Traceback (most recent call last):
        8: from /usr/local/bin/github-linguist:23:in `<main>'
        7: from /usr/local/bin/github-linguist:23:in `load'
        6: from /var/lib/gems/2.7.0/gems/github-linguist-7.27.0/bin/github-linguist:152:in `<top (required)>'
        5: from /var/lib/gems/2.7.0/gems/github-linguist-7.27.0/bin/github-linguist:109:in `github_linguist'
        4: from /var/lib/gems/2.7.0/gems/github-linguist-7.27.0/lib/linguist/blob_helper.rb:155:in `text?'
        3: from /var/lib/gems/2.7.0/gems/github-linguist-7.27.0/lib/linguist/blob_helper.rb:127:in `binary?'
        2: from /var/lib/gems/2.7.0/gems/github-linguist-7.27.0/lib/linguist/lazy_blob.rb:83:in `data'
        1: from /var/lib/gems/2.7.0/gems/github-linguist-7.27.0/lib/linguist/lazy_blob.rb:109:in `load_blob!'
/var/lib/gems/2.7.0/gems/github-linguist-7.27.0/lib/linguist/lazy_blob.rb:109:in `to_buffer': wrong argument type nil (expected String) (TypeError)

a simple reproduce repo https://github.com/Icyrockton/linguistic_test

python main.py

[lildude]

Your repo link is inaccessible.

can I force a bypass of the .gitignore restriction?

No. Linguist expects to work on files in a git repo by default and will always look first to see if the file is in a repo. If the files are in a repo but in a directory ignored by git, then Linguist (technically rugged which does the repo accessing etc) won't be able to find the files and you get the behaviour you are seeing.

This is the section of code doing this (lines 97-104):

linguist/bin/github-linguist

Lines 97 to 107 in 7ca3799

	begin
	# Check if this file is inside a git repository so we have things like
	# `.gitattributes` applied.
	file_full_path = File.realpath(path)
	rugged = Rugged::Repository.discover(file_full_path)
	file_rel_path = file_full_path.sub(rugged.workdir, '')
	oid = -> { rugged.head.target.tree.walk_blobs { |r, b| return b[:oid] if r + b[:name] == file_rel_path } }
	blob = Linguist::LazyBlob.new(rugged, oid.call, file_rel_path)
	rescue Rugged::RepositoryError
	blob = Linguist::FileBlob.new(path, Dir.pwd)
	end

The only way around this is to put the files in a directory completely outside of any git repo which will then be taken care of by line 106 above.

[Icyrockton]

My mistake! I have changed the visibility to public.
Since the downloaded files are at the 100GB level, redundant copy is very trivial, I just wanted to use lingstuic's file language detection feature, which is the best library I've found to use.
Thanks for your reply by the way! :)

[lildude]

If you're not tied to using Ruby, you might be interested in go-enry. This is written in Go and feeds directly off Linguist and is what GitHub's Search functionality uses. I'm not sure if it has the same repo requirements and limitations, but it's probably worth a look.

[Icyrockton]

Thank you for your help, I will check it out ❤️