mirror of
https://github.com/KingDuckZ/dindexer.git
synced 2025-02-19 12:04:54 +00:00
Trim down lengthy stuff from the readme.
Make it more clear, more up-to-date and less verbose. Hopefully.
This commit is contained in:
parent
6a03afb938
commit
c61dc4fd9c
1 changed files with 101 additions and 53 deletions
154
README.md
154
README.md
|
|
@ -1,55 +1,48 @@
|
|||
# dindexer #
|
||||
## Project info and status ##
|
||||
### Quick project info ###
|
||||
#### Release build status ####
|
||||
[TOC]
|
||||
# dindexer #
|
||||
## Project info ##
|
||||
### Release build status ###
|
||||
[](https://drone.io/bitbucket.org/King_DuckZ/dindexer/latest)
|
||||
|
||||
#### Latest release ###
|
||||
### Latest release ##
|
||||
Latest stable release is __0.1.5b__.
|
||||
|
||||
#### Flattr ####
|
||||
### Flattr ###
|
||||
[](https://flattr.com/submit/auto?user_id=King_DuckZ&url=https%3A%2F%2Fbitbucket.org%2FKing_DuckZ%2Fdindexer&title=dindexer&language=en_GB&tags=bitbucket&category=software)
|
||||
|
||||
Please support the development of this software if you like it!
|
||||
|
||||
### Development news ###
|
||||
### News ###
|
||||
[](https://mondiaspora.net/tags/dindexer)
|
||||
Follow [my posts on Diaspora](https://mondiaspora.net/tags/dindexer) for the latest development news and to post your comments about the project!
|
||||
|
||||
## What is dindexer? ##
|
||||
### Purpose ###
|
||||
dindexer (from "Disc Indexer") is a program to help you keep track of where your files are as you back them up on external media. It's like updatedb/locate, but it's thought for never changing removable media, such as DVDs and BluRays.
|
||||
## What is dindexer? ##
|
||||
### Purpose ###
|
||||
dindexer (from "Disc Indexer") is a program to help you keep track of where your files are as you back them up on external media. It's like updatedb/locate, but it's designed for never changing removable media, such as DVDs and BluRays.
|
||||
|
||||
Running dindexer on your disks will build a list of all your files and directories. You can associate those lists to a descriptive name and a number, so it's easy to find on what disk a file is on the next time you search for that file.
|
||||
|
||||
dindexer will also help you keeping track of duplicate data.
|
||||
|
||||
### What dindexer is not? ###
|
||||
### What dindexer is not? ###
|
||||
To further clarify what dindexer's intended usage is, let's make a list of what it *won't* do for you:
|
||||
|
||||
* it's not a daemon - it's just a regular program that starts when you invoke it, performs its duties and then quits
|
||||
* it's not a realtime file indexing utility; I believe there are a few indexing tools out there already, such as Baloo, mlocate and surely others, but dindexer doesn't duplicate any of those, nor wants to replace them
|
||||
* it's not a backup tool; dindexer won't backup anything at all, won't make any copies of anything nor anything like that, so it's not like rsync; if you lose your DVD you previously scanned with dindexer, sorry but you've lost all of the data on your DVD
|
||||
|
||||
### Usage example ###
|
||||
You have a pile of DVDs, BluRays or any other external storage media on which you stored a copy of your files. Maybe you wanted to keep YouTube videos you don't want to have on your hard disk anymore, your pictures from your trip to Portugal a few years ago, your gpg private key backup and your Humble Bundle games. Maybe you're just moving away old stuff from your external hard disk onto DVDs, so you can store new data on your hard disk.
|
||||
### Use case ###
|
||||
You have a pile of DVDs and BluRays holding your files. Maybe you wanted to keep that funny YouTube video, your pictures from your trip to Portugal, your gpg private key backup and your Humble Bundle games.
|
||||
|
||||
It's been a few years since you burned your first DVD, and your collection of backup has only grown more. You got to the point it's hard to tell what you have on your DVDs (especially those you just marked as "Backup March 2012"). Even worse, you don't know if those pictures from Portugal are already on some DVD - you think you backed them up somewhere, but just to be safe you burn them again on the next "Backup December 2015" DVD.
|
||||
Your home directory is full again, and it's time to make a new backup DVD. How can you make sure that those files are not on some DVD already? What can you just delete and what needs to be backed up first? This is where dindexer can help you.
|
||||
|
||||
Five months later, you want to show your pictures of your trip to Portugal to your friend, but you finally deleted them from your hard disk. You probably have at least two copies on your DVDs, but where exactly? As you browse through your "pics backup"-labeled DVD and you realize it's a completely different sort of pictures you have on there, you think you should just give up on your Portugal photos for now.
|
||||
### Usage flow example ###
|
||||
|
||||
dindexer aims to save you from that scenario. The intended usage flow is:
|
||||
|
||||
1. Prepare the list of files you want to burn in an external program like k3b for example, and make your backup
|
||||
1. Prepare the list of files you want to burn in an external program like k3b for example, and burn your disc
|
||||
2. When finished, mount your new backup, for example on /media/cdrom
|
||||
3. Scan your backup with dindexer using `dindexer scan --setname "Disc 24 - pictures and stuff" /media/cdrom`
|
||||
4. When done, the list of all the files on your disc will be saved in dindexer's database
|
||||
5. Make queries to find on which disc you stored your data - this part is still being worked on, but you can imagine something like `dindexer query portugal.jpg` returning "Disc 24 - pictures and stuff"
|
||||
6. Go to where you dump - er... store your physical discs (the drawer in your desk, a box, the other room), find the disc you marked as "Disc 24", pop it in your cd-reader and voilà, here's where your portugal.jpg was!
|
||||
4. When done, the list of all the files on your disc will be saved in the database
|
||||
5. Make queries to find on which disc you stored your data `dindexer locate portugal.jpg`
|
||||
|
||||
**WARNING:** as dindexer is still being heavily developed, any of those steps can change in the future. I have a few ideas on how to make this program more convenient to use, but this will require some work. Contributions are welcome.
|
||||
|
||||
### Configuration ###
|
||||
### Configuration ###
|
||||
As dindexer needs to write data to a PostgreSQL database, you need to configure access to it. Create a ~/.config/dindexer.yml file with the settings to connect to your database. For example:
|
||||
|
||||
%YAML 1.2
|
||||
|
|
@ -61,25 +54,59 @@ As dindexer needs to write data to a PostgreSQL database, you need to configure
|
|||
port: 5432
|
||||
address: 127.0.0.1
|
||||
|
||||
### Usage ###
|
||||
Using dindexer is pretty straightforward: mount your DVD and scan it. For example:
|
||||
## Actions ##
|
||||
Actions are just executable files located in a directory that is searched by dindexer. dindexer will list available actions when invoked with no parameters.
|
||||
|
||||
mount /dev/cdrom /media/my_backup
|
||||
dindexer scan --setname "My generic backup 2015-11-13" /media/my_backup
|
||||
To get help on any specific action you can use the --help switch:
|
||||
|
||||
dindexer <action> --help
|
||||
|
||||
For example:
|
||||
|
||||
dindexer scan --help
|
||||
|
||||
### scan ###
|
||||
The program will go through every file in the path you specify. Hashing everything in that path could take a bit of time, so please be patient.
|
||||
When dindexer is done analyzing your directory it will save everything to the db in one go, unless the same item was already present in the db.
|
||||
|
||||
You can run dindexer --help to see a list of available switches.
|
||||
To get a list of all commands you can just run `dindexer --help`. To get help on a specific command, run `dindexer <command> --help`.
|
||||
If you are scanning a defective disc, you can use the `--ignore-errors`. dindexer will then skip unreadable files and mark them as such in the db.
|
||||
|
||||
### Code architecture ###
|
||||
I wrote dindexer with the idea of having at least one GUI for it at som point.
|
||||
Since I'm just writing a CLI interface for now, I've decided to put all of the 'core' functionalities in a shared object (*machinery*). Other useful, lower level bits can be found in *common*.
|
||||
Example
|
||||
|
||||
As the code is being changed all the time, I'll be waiting to get a more stable interface before writing any detailed documentation. In the meantime you can have a look at the various command line programs to find examples of how to use the different parts of machinery, or you can get in touch with me on IRC (see *Contact me* at the bottom).
|
||||
dindexer scan --setname "My backup 2015-11-13" /media/my_backup
|
||||
|
||||
## Build instructions ##
|
||||
### Dependencies ###
|
||||
### delete ###
|
||||
Delete sets from your db by ID. You can obtain the ID of a set using the locate command.
|
||||
|
||||
Example
|
||||
|
||||
#This will delete sets 3 and 28 if they exist
|
||||
dindexer delete 3 28
|
||||
|
||||
### locate ###
|
||||
Find files in the db, similarly to the `locate` command in Linux. Pass the `--set` or `-s` switch to look for sets instead.
|
||||
|
||||
Example
|
||||
|
||||
dindexer locate picture.jpg
|
||||
|
||||
### navigate ###
|
||||
Work in progress. Provides a bash-like CLI so you can explore your sets as if you had put your CDs in the reader (except that you won't be able to read any file data). At the moment only a few commands are recognized, such as `ls`, `pwd` and `cd`.
|
||||
|
||||
Example
|
||||
|
||||
dindexer navigate
|
||||
|
||||
### tag ###
|
||||
Allows you to tag single files or whole sets for easier later retrieval. Also use this command to remove tags (see action help).
|
||||
|
||||
Example
|
||||
|
||||
#tag all ogg and flac files in set 31 as music and jpop
|
||||
dindexer tag music,jpop --set 31 '*.ogg' '*.flac'
|
||||
|
||||
## Build instructions ##
|
||||
### Dependencies ###
|
||||
The following libraries must be available on your system:
|
||||
|
||||
- PostgreSQL (libpq)
|
||||
|
|
@ -89,7 +116,21 @@ The following libraries must be available on your system:
|
|||
|
||||
*Note:* Although the CMakeLists.txt will say minimum required version for PostgreSQL is 8.3, this program has only ever been tested with 9.4. 8.3 is simply the threshold under which I'm sure the code will not work. If you want to try dindexer with a PostgreSQL version less than 9.4 you are welcome to do so, but please note I won't be able to make sure everything will work. Patches that address eventual issues are welcome.
|
||||
|
||||
### Linux ###
|
||||
### Build options ###
|
||||
These are the options understood by cmake. At runtime you might be able to see what *most* of those options were set to using `dindexer --builtin`.
|
||||
|
||||
* **DB_OWNER_NAME** Default user name for backends that require one
|
||||
* **DINDEXER_ACTIONS_PATH** Search path for dindexer actions
|
||||
* **DINDEXER_CONFIG_FILE** Full path to the yaml configuration file
|
||||
* **DINDEXER_CXX11_ABI** Set this to off to force newer gcc (5+) to keep using the old ABI - useful on Gentoo if you built your system libraries with gcc 4 and you are trying to build dindexer with gcc 5
|
||||
* **DINDEXER_DEBUG_CFG_FILE** If set to on, DINDEXER_CONFIG_FILE will be set to a path that is more appropriate for development environments
|
||||
* **DINDEXER_NATIVE_RELEASE** Set to on to pass `--march=native` to the compiler
|
||||
* **DINDEXER_WITH_BUILD_DATE** If set to on, the current date will be hardcoded into the final binary - warning: if set to on, some files will need to be rebuilt every day just because the date has changed
|
||||
* **DINDEXER_WITH_MEDIA_AUTODETECT** Enable code that detects the inserted media type, eg: CD-Rom or DVD - requires libblkid
|
||||
* **DINDEXER_WITH_NICE_MEDIA_TYPES** If DINDEXER_WITH_MEDIA_AUTODETECT is enabled, this option will print slightly nicer names for media types (for example in the output of `dindexer scan --help`) - requires C++14
|
||||
* **PBL_WITH_TESTS** When set to on (default when you set BUILD_TESTING to on) it will cause unit tests from the pbl library to be also built - you can safely disable this, as some tests are really slow and the pbl project is not changing anymore
|
||||
|
||||
### Linux ###
|
||||
If you are a regular user and want to make a release build for yourself:
|
||||
|
||||
mkdir dindexer_build
|
||||
|
|
@ -109,11 +150,15 @@ If you plan to hack on the code and do some development instead:
|
|||
|
||||
Binaries will be in the src directory. Use `make install` to install them to your selected install prefix (`-CMAKE_INSTALL_PREFIX:PATH=<your_path>` option in CMake). For development purposes, I create a symlink to the main program in the root of my build directory with the command `ln -s src/main/dindexer`, so I can run the various subcommands from the top-level build directory directly (eg: `./dindexer navigate`).
|
||||
|
||||
### Other platforms ###
|
||||
### Other platforms ###
|
||||
I never tested dindexer on anything other than Linux. Note that the code assumes paths to be Unix-like. It also makes other assumptions about the current platform being Linux, so porting could be not so straightforward.
|
||||
|
||||
## Useful technical details ##
|
||||
### Program overview ###
|
||||
## Useful technical details ##
|
||||
### Code architecture ###
|
||||
All of the core dindexer functionality is in the *machinery* project. Other useful bits can be found in *common*. Note that *common* depends on *machinery*.
|
||||
If you want to write your own interface to dindexer, you probably only need to take *machinery* and its dependencies. If you are writing a command line program, stuff in *common* could also be useful to you.
|
||||
|
||||
### Program overview ###
|
||||
For the sake of clarity, from here on I will refer to the main program (more on it later) simply as "dindexer" and collectively call all the other executables "actions". I will refer to static and shared objects as "libraries".
|
||||
"dindexer" is also the name of this project as a whole; that meaning will be kept and it will hopefully be clear from the context.
|
||||
|
||||
|
|
@ -125,6 +170,7 @@ Other than dindexer, in src/ you can find the following actions:
|
|||
* **locate**
|
||||
* **query**
|
||||
* **scan**
|
||||
* **tag**
|
||||
|
||||
and the following libraries:
|
||||
|
||||
|
|
@ -137,9 +183,9 @@ Additionally inside include/helpers/ you will find header-only implementations o
|
|||
Currently, actions are just the command line frontend to access the functionalities in the libraries. This way writing a new CLI or GUI should be relatively easy.
|
||||
This structure might change in the future.
|
||||
|
||||
### Queries ###
|
||||
### Queries ###
|
||||
|
||||
#### Find duplicates ####
|
||||
#### Find duplicates ####
|
||||
List all duplicate files belonging to different sets:
|
||||
|
||||
select files.hash, group_id, t.ct from files
|
||||
|
|
@ -149,10 +195,10 @@ List all duplicate files belonging to different sets:
|
|||
on t.hash = files.hash
|
||||
group by files.hash, group_id, t.ct order by files.hash;
|
||||
|
||||
### Set number ###
|
||||
### Set number ###
|
||||
In the sets table you can find a disk_number column that is currently not used. This is in case you have numbered discs, so you are free to put any number you like in that column.
|
||||
|
||||
### Disc type ###
|
||||
### Disc type ###
|
||||
For your convenience, you can store the type of the disc you are going to index. Currently this has no impact on the program's behaviour. Available types are:
|
||||
|
||||
* **C** - CD-Rom
|
||||
|
|
@ -164,16 +210,18 @@ For your convenience, you can store the type of the disc you are going to index.
|
|||
* **Z** - Iomega Zip
|
||||
* **O** - Other
|
||||
|
||||
## Future development ##
|
||||
## Future development ##
|
||||
This section contains simple ideas for features I'd like to add in the future.
|
||||
|
||||
### Project name ###
|
||||
### Project name ###
|
||||
Although "dindexer" sounds like a very intuitive name for what this program does, I'm not entirely satisfied with the name. I'm open to suggestions for a new name.
|
||||
|
||||
### Autosetup ###
|
||||
Currently `catafy` (from "catalog" + "file") is the only alternative name that has been proposed.
|
||||
|
||||
### Autosetup ###
|
||||
When you first run dindexer you have to make sure the database already exists and the tables have already been created. While this is somewhat fine for developers, and you get an auto-generated sql script in your build directory to make this step as easy as possible, I think it should be completely automated so first time users can just start dindexer from the very first time.
|
||||
|
||||
### Database ###
|
||||
### Database ###
|
||||
Currently dindexer only works with PostgreSQL databases. This will sound like a very heavy dependency, but having to access my database from multiple machines I thought this would be the best backend to start from. In my case the database is running on a raspberry pi.
|
||||
|
||||
In the future I'd like to add support for other storage backends, such as:
|
||||
|
|
@ -183,14 +231,14 @@ In the future I'd like to add support for other storage backends, such as:
|
|||
* Redis (?)
|
||||
* ...and possibly others, depending on how much help I get from the community
|
||||
|
||||
### Multiusers ###
|
||||
### Multiusers ###
|
||||
As it is, dindexer can already be used by multiple users at the same time, as it relies on PostgreSQL. It would be nice however to have some other features, such as what user scanned a given disc and from which machine.
|
||||
|
||||
### GUI ###
|
||||
### GUI ###
|
||||
I've been developing this program with GUIs in mind. However, since I don't have much experience with GUI designing and programming, I decided to leave this for later and go for a more straightforward CLI. This is definitely an area I'd like to get help with.
|
||||
|
||||
While I was tempted to look into QT5, some people on IRC suggested I write a Dolphin plugin instead of a full-blown GUI program. Neither option exclude the other, since all the core functionality is in shared libraries. In fact if you want to program your own GUI around dindexer you just need to pass the user's input on to the right function.
|
||||
Or at least that's the idea behind the design of the code :)
|
||||
|
||||
## Contact me ##
|
||||
## Contact me ##
|
||||
The best way to get in touch is to find me on IRC freenode. You can find me as either King_DuckZ, or you can try Ccdc_DuckZ if you get any afk autoreply.
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue