Fetch files with HTTP

Files can be fetched with HTTP from files.unifaun.se. You can either use HTTP or HTTPS but HTTPS is recommended.

There are three methods of authentication. The authentication method is specified with the “auth” parameter. If the “auth” parameter is omitted and no “Authorization” HTTP header is sent, the authentication is set to HTTP Digest.

HTTP Digest access authentication

The default authentication mode. The advantage is that the password isn't transmitted unmodified to the web server which makes it safer than basic authentication.

Set the parameter “auth” to “DIGEST” or don't send the parameter.

Important

User name should be prefixed with “UO.”, e.g. UO.002000111.

Please refer to https://tools.ietf.org/html/rfc2617.

HTTP Basic access authentication

User name and password are sent in the HTTP header. This method is only safe if HTTPS is used.

Set the parameter “auth” to “BASIC”. The server will also try to automatically detect this authentication method using the HTTP header “Authorization”.

Please refer to https://www.w3schools.com/tags/ref_urlencode.asp.

Query string parameters

Send in credentials as on the query string of the URL. This method is only safe if HTTPS is used.

Set the parameter “auth” to “URL”. Use the parameters “authusr” and “authpwd” for user and password.

Example: https://files.unifaun/directory/file.txt?auth=URL&authusr=user&authpwd=password

Fetching files

List the files in a directory
Use an HTTP GET with the directory path as URL. The parameter view=SIMPLETEXT returns a result in plain text that is easy to interpret. URLs should use UTF-8 character set encoding and standard “URL encoding”. Each row in the result represents a file and is terminated by a line feed (0x0A). The encoding of the result is UTF-8.
```
GET /files/?view=SIMPLETEXT HTTP/1.1

File1.txt
File2.txt
File3.txt
```
Fetch files
Use an HTTP GET with the file path as URL.
```
GET /files/File1.txt?view=SIMPLETEXT HTTP/1.1
```
Delete files
Use an HTTP DELETE with the file path as URL.
```
DELETE /files/File1.txt?view=SIMPLETEXT HTTP/1.1
```
Another method for deleting the file is to use an HTTP POST with the file path as URL and a parameter “action” with the value “delete”:
```
POST /files/File1.txt?view=SIMPLETEXT&action=delete HTTP/1.1
```

Using Wget

GNU Wget is a command line utility that is available both for Microsoft Windows and Unix-like operating systems.

Please refer to http://www.gnu.org/software/wget/ and http://gnuwin32.sourceforge.net/packages/wget.htm. The following script for Unix (bash shell) fetches all files that end with “.txt” from a directory and deletes the files from the source.

#!/bin/bash
HOST='user:password@files.unifaun.se'
DIRECTORY='out'
PARAMS='view=SIMPLETEXT'
FILES=$(wget -qO - "https://$HOST/$DIRECTORY/?$PARAMS" | grep -i ".txt$")
echo "Fetching files:" $FILES
for FILE in $FILES; do
  wget -qO - "https://$HOST/$DIRECTORY/$FILE" > $FILE && \
  wget –qO - "https://$HOST/$DIRECTORY/$FILE" \
    --post-data='action=delete' > /dev/null;
done

The command wget -nc -nd -r -l1 --no-parent -A "*.txt " \ will fetch all files in the “out” directory that end with “.txt” and haven't already been fetched.

https://user:pwd@files.unifaun.se/out/?view=SIMPLEHTML

Using curl

Curl is a command line utility that is available on several operating systems. Please refer to http://curl.haxx.se/.

The following script for Unix (Bash shell) fetches all files that end with “.txt” from a directory and deletes the files from the source.

#!/bin/bash
HOST='user:password@files.unifaun.se'
DIRECTORY='out'
PARAMS='view=SIMPLETEXT'
FILES=$(curl -sS "https://$HOST/$DIRECTORY/?$PARAMS" | grep -i ".txt$")
echo "Fetching files:" $FILES
for FILE in $FILES; do
  curl -OfsS "https://$HOST/$DIRECTORY/$FILE" && \
  curl -fsS "https://$HOST/$DIRECTORY/$FILE" \
    --form-string "action=delete" > /dev/null;
done

The following script for Windows (Powershell) fetches all files that end with “.txt” from a directory and deletes the files from the source.

$TARGETHOST="user:password@files.unifaun.se"
$DIRECTORY="out"
$PARAMS="view=SIMPLETEXT"
$FILES = @(curl -sS "https://$TARGETHOST/$DIRECTORY/?$PARAMS" `
  | select-string -pattern "\.txt$")
if ($FILES.length -gt 0) {
  write-host "Fetching files:" $FILES
  foreach ($FILE in $FILES) {
    curl -OfsS https://$TARGETHOST/$DIRECTORY/$FILE
    if ($?) {
      curl -fsS "https://$TARGETHOST/$DIRECTORY/$FILE" `
        --form-string "action=delete" > $null
    }
  }
}

In this section:

Unifaun Online