Bump version

* Avoid stall on web scraping

This patch fixes stall on web scraping.
I encountered a stall (scraping never ends) when adding
a bookmark of some site.
To avoid this case, adding a timeout parameter at requests.get()
function is a solution.

Signed-off-by: Taku Izumi <admin@orz-style.com>

* Avoid character corruption of scraping some Japanese sites

This patch fixes character corruption of scraping some Japanese
sites. To avoid character corruption, I use r.content instead
of r.text in load_page function.

The reason of character corruption is encoding problem, I think.
r.text handles data as unicode encoded text, so if scraping
web site's charset is not unicode encoded, character corruption
occurs. r.content handles data as str[], we can avoid encoding
problem.

Signed-off-by: Taku Izumi <admin@orz-style.com>

* use charset_normalizer to determine response encoding

Co-authored-by: Taku Izumi <admin@orz-style.com>
Co-authored-by: Sascha Ißbrücker <sascha.issbruecker@googlemail.com>

CHANGELOG.md

@@ -1,5 +1,22 @@
 # Changelog
 
+## v1.7.1 (25/08/2021)
+- [**bug**] umlaut/non-ascii characters broken when using bookmarklet (firefox) [#148](https://github.com/sissbruecker/linkding/issues/148)
+- [**bug**] Bookmark import accepts empty URL values [#124](https://github.com/sissbruecker/linkding/issues/124)
+- [**enhancement**] Show the version in the settings [#104](https://github.com/sissbruecker/linkding/issues/104)
+
+---
+
+## v1.7.0 (17/08/2021)
+- Upgrade to Django 3
+- Bump other dependencies
+
+---
+
+## v1.6.5 (15/08/2021)
+- [**enhancement**] query with multiple hashtags very slow [#112](https://github.com/sissbruecker/linkding/issues/112)
+---
+
 ## v1.6.4 (13/05/2021)
 - Update dependencies for security fixes
 

Dockerfile

@@ -47,6 +47,8 @@ EXPOSE 9090
 # Activate virtual env
 ENV VIRTUAL_ENV /opt/venv
 ENV PATH /opt/venv/bin:$PATH
+# Allow running containers as an an arbitrary user in the root group, to support deployment scenarios like OpenShift, Podman
+RUN ["chmod", "g+w", "."]
 # Run bootstrap logic
 RUN ["chmod", "+x", "./bootstrap.sh"]
 CMD ["./bootstrap.sh"]

README.md

@@ -88,7 +88,7 @@ If you can not or don't want to use Docker you can install the application manua
 
 ### Hosting
 
-The application runs in a web-server called [uWSGI](https://uwsgi-docs.readthedocs.io/en/latest/) that is production-ready and that you can expose to the web. If you don't know how to configure your server to expose the application to the web there are several more steps involved. I can not support support the process here, but I can give some pointers on what to search for:
+The application runs in a web-server called [uWSGI](https://uwsgi-docs.readthedocs.io/en/latest/) that is production-ready and that you can expose to the web. If you don't know how to configure your server to expose the application to the web there are several more steps involved. I can not support the process here, but I can give some pointers on what to search for:
 - first get the app running (described in this document)
 - open the port that the application is running on in your servers firewall
 - depending on your network configuration, forward the opened port in your network router, so that the application can be addressed from the internet using your public IP address and the opened port
@@ -126,7 +126,7 @@ Note that any proxy servers that you are running in front of linkding may have t
 
 ## Development
 
-The application is open source, so you are free to modify or contribute. The application is built using the Django web framework. You can get started by checking out the excellent Django docs: https://docs.djangoproject.com/en/3.0/. The `bookmarks` folder contains the actual bookmark application, `siteroot` is the Django root application. Other than that the code should be self-explanatory / standard Django stuff 🙂.
+The application is open source, so you are free to modify or contribute. The application is built using the Django web framework. You can get started by checking out the excellent Django docs: https://docs.djangoproject.com/en/3.2/. The `bookmarks` folder contains the actual bookmark application, `siteroot` is the Django root application. Other than that the code should be self-explanatory / standard Django stuff 🙂.
 
 ### Prerequisites
 - Python 3

bookmarks/admin.py

@@ -5,7 +5,7 @@ from django.contrib.auth.models import User
 from django.db.models import Count, QuerySet
 from django.utils.translation import ngettext, gettext
 from rest_framework.authtoken.admin import TokenAdmin
-from rest_framework.authtoken.models import Token
+from rest_framework.authtoken.models import TokenProxy
 
 from bookmarks.models import Bookmark, Tag, UserProfile
 from bookmarks.services.bookmarks import archive_bookmark, unarchive_bookmark
@@ -97,4 +97,4 @@ linkding_admin_site = LinkdingAdminSite()
 linkding_admin_site.register(Bookmark, AdminBookmark)
 linkding_admin_site.register(Tag, AdminTag)
 linkding_admin_site.register(User, AdminCustomUser)
-linkding_admin_site.register(Token, TokenAdmin)
+linkding_admin_site.register(TokenProxy, TokenAdmin)

bookmarks/services/importer.py

@@ -1,6 +1,5 @@
 import logging
 from dataclasses import dataclass
-from datetime import datetime
 
 from django.contrib.auth.models import User
 from django.utils import timezone
@@ -8,6 +7,7 @@ from django.utils import timezone
 from bookmarks.models import Bookmark, parse_tag_string
 from bookmarks.services.parser import parse, NetscapeBookmark
 from bookmarks.services.tags import get_or_create_tags
+from bookmarks.utils import parse_timestamp
 
 logger = logging.getLogger(__name__)
 
@@ -47,7 +47,7 @@ def _import_bookmark_tag(netscape_bookmark: NetscapeBookmark, user: User):
 
     bookmark.url = netscape_bookmark.href
     if netscape_bookmark.date_added:
-        bookmark.date_added = datetime.utcfromtimestamp(int(netscape_bookmark.date_added)).astimezone()
+        bookmark.date_added = parse_timestamp(netscape_bookmark.date_added)
     else:
         bookmark.date_added = timezone.now()
     bookmark.date_modified = bookmark.date_added
@@ -57,6 +57,7 @@ def _import_bookmark_tag(netscape_bookmark: NetscapeBookmark, user: User):
         bookmark.description = netscape_bookmark.description
     bookmark.owner = user
 
+    bookmark.full_clean()
     bookmark.save()
 
     # Set tags

bookmarks/services/website_loader.py

@@ -2,6 +2,7 @@ from dataclasses import dataclass
 
 import requests
 from bs4 import BeautifulSoup
+from charset_normalizer import from_bytes
 
 
 @dataclass
@@ -33,5 +34,11 @@ def load_website_metadata(url: str):
 
 
 def load_page(url: str):
-    r = requests.get(url)
-    return r.text
+    r = requests.get(url, timeout=10)
+
+    # Use charset_normalizer to determine encoding that best matches the response content
+    # Several sites seem to specify the response encoding incorrectly, so we ignore it and use custom logic instead
+    # This is different from Response.text which does respect the encoding specified in the response first,
+    # before trying to determine one
+    results = from_bytes(r.content)
+    return str(results.best())

bookmarks/templates/settings/api.html

@@ -17,7 +17,7 @@
             </div>
             <p><strong>Please treat this token as you would any other credential.</strong> Any party with access to this
                 token can access and manage all your bookmarks.</p>
-            <p>If you think that a token was compromised you can revoke (delete) it in the <a href="{% url 'admin:authtoken_token_changelist' %}">admin panel</a>. After deleting the token, a new one will be generated when you reload this settings page.</p>
+            <p>If you think that a token was compromised you can revoke (delete) it in the <a href="{% url 'admin:authtoken_tokenproxy_changelist' %}">admin panel</a>. After deleting the token, a new one will be generated when you reload this settings page.</p>
         </section>
 
     </div>

bookmarks/templates/settings/general.html

@@ -69,6 +69,15 @@
             {% endif %}
         </section>
 
+        {# About section #}
+        <section class="content-area">
+            <h2>About</h2>
+            <p>Version: {{ app_version }}</p>
+            <p>
+                Code: <a href="https://github.com/sissbruecker/linkding/"
+                    target="_blank">GitHub</a>
+            </p>
+        </section>
     </div>
 
 {% endblock %}

bookmarks/tests/helpers.py

@@ -1,4 +1,5 @@
 import random
+import logging
 
 from django.contrib.auth.models import User
 from django.utils import timezone
@@ -117,3 +118,14 @@ def random_sentence(num_words: int = None, including_word: str = ''):
     random.shuffle(selected_words)
 
     return ' '.join(selected_words)
+
+
+def disable_logging(f):
+    def wrapper(*args):
+        logging.disable(logging.CRITICAL)
+        result = f(*args)
+        logging.disable(logging.NOTSET)
+
+        return result
+
+    return wrapper

bookmarks/tests/resources/simple_valid_import_file.html

@@ -11,10 +11,10 @@
     <DT><A HREF="https://example.com/1" ADD_DATE="1616337559" PRIVATE="0" TOREAD="0" TAGS="tag1">test title 1</A>
     <DD>test description 1
 
-    <DT><A HREF="https://example.com/2" ADD_DATE="1616337559" PRIVATE="0" TOREAD="0" TAGS="tag2">test title 2</A>
+    <DT><A HREF="https://example.com/2" ADD_DATE="1616337559000" PRIVATE="0" TOREAD="0" TAGS="tag2">test title 2</A>
     <DD>test description 2
 
-    <DT><A HREF="https://example.com/3" ADD_DATE="1616337559" PRIVATE="0" TOREAD="0" TAGS="tag3">test title 3</A>
+    <DT><A HREF="https://example.com/3" ADD_DATE="1616337559000000" PRIVATE="0" TOREAD="0" TAGS="tag3">test title 3</A>
     <DD>test description 3
 
 </DL><p>

bookmarks/tests/test_bookmark_validation.py

@@ -34,6 +34,27 @@ class BookmarkValidationTestCase(TestCase):
     def setUp(self) -> None:
         self.user = User.objects.create_user('testuser', 'test@example.com', 'password123')
 
+    def test_bookmark_model_should_not_allow_missing_url(self):
+        bookmark = Bookmark(
+            date_added=datetime.datetime.now(),
+            date_modified=datetime.datetime.now(),
+            owner=self.user
+        )
+
+        with self.assertRaises(ValidationError):
+            bookmark.full_clean()
+
+    def test_bookmark_model_should_not_allow_empty_url(self):
+        bookmark = Bookmark(
+            url='',
+            date_added=datetime.datetime.now(),
+            date_modified=datetime.datetime.now(),
+            owner=self.user
+        )
+
+        with self.assertRaises(ValidationError):
+            bookmark.full_clean()
+
     @override_settings(LD_DISABLE_URL_VALIDATION=False)
     def test_bookmark_model_should_validate_url_if_not_disabled_in_settings(self):
         self._run_bookmark_model_url_validity_checks(ENABLED_URL_VALIDATION_TEST_CASES)

bookmarks/tests/test_importer.py

@@ -0,0 +1,33 @@
+from django.test import TestCase
+
+from bookmarks.services.importer import import_netscape_html
+from bookmarks.tests.helpers import BookmarkFactoryMixin, disable_logging
+
+
+class ImporterTestCase(TestCase, BookmarkFactoryMixin):
+
+    def create_import_html(self, bookmark_tags_string: str):
+        return f'''
+<!DOCTYPE NETSCAPE-Bookmark-file-1>
+<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=UTF-8">
+<TITLE>Bookmarks</TITLE>
+<H1>Bookmarks</H1>
+<DL><p>
+{bookmark_tags_string}
+</DL><p>
+        '''
+
+    @disable_logging
+    def test_validate_empty_or_missing_bookmark_url(self):
+        test_html = self.create_import_html(f'''
+            <!-- Empty URL -->
+            <DT><A HREF="" ADD_DATE="1616337559" PRIVATE="0" TOREAD="0" TAGS="tag3">Empty URL</A>
+            <DD>Empty URL
+            <!-- Missing URL -->
+            <DT><A ADD_DATE="1616337559" PRIVATE="0" TOREAD="0" TAGS="tag3">Missing URL</A>
+            <DD>Missing URL
+        ''')
+
+        import_result = import_netscape_html(test_html, self.get_or_create_test_user())
+
+        self.assertEqual(import_result.success, 0)

bookmarks/tests/test_settings_import_view.py

@@ -1,7 +1,7 @@
 from django.test import TestCase
 from django.urls import reverse
 
-from bookmarks.tests.helpers import BookmarkFactoryMixin
+from bookmarks.tests.helpers import BookmarkFactoryMixin, disable_logging
 
 
 class SettingsImportViewTestCase(TestCase, BookmarkFactoryMixin):
@@ -52,6 +52,7 @@ class SettingsImportViewTestCase(TestCase, BookmarkFactoryMixin):
         self.assertNoFormSuccessHint(response)
         self.assertFormErrorHint(response, 'Please select a file to import.')
 
+    @disable_logging
     def test_should_show_hint_if_import_raises_exception(self):
         with open('bookmarks/tests/resources/invalid_import_file.png', 'rb') as import_file:
             response = self.client.post(
@@ -64,6 +65,7 @@ class SettingsImportViewTestCase(TestCase, BookmarkFactoryMixin):
             self.assertNoFormSuccessHint(response)
             self.assertFormErrorHint(response, 'An error occurred during bookmark import.')
 
+    @disable_logging
     def test_should_show_respective_hints_if_not_all_bookmarks_were_imported_successfully(self):
         with open('bookmarks/tests/resources/simple_valid_import_file_with_one_invalid_bookmark.html') as import_file:
             response = self.client.post(

bookmarks/tests/test_utils.py

@@ -1,9 +1,10 @@
 from unittest.mock import patch
 
+from dateutil.relativedelta import relativedelta
 from django.test import TestCase
 from django.utils import timezone
 
-from bookmarks.utils import humanize_absolute_date, humanize_relative_date
+from bookmarks.utils import humanize_absolute_date, humanize_relative_date, parse_timestamp
 
 
 class UtilsTestCase(TestCase):
@@ -63,3 +64,45 @@ class UtilsTestCase(TestCase):
         # Regression: Test that subsequent calls use current date instead of cached date (#107)
         with patch.object(timezone, 'now', return_value=timezone.datetime(2021, 1, 13)):
             self.assertEqual(humanize_relative_date(timezone.datetime(2021, 1, 13)), 'Today')
+
+    def verify_timestamp(self, date, factor=1):
+        timestamp_string = str(int(date.timestamp() * factor))
+        parsed_date = parse_timestamp(timestamp_string)
+        self.assertEqual(date, parsed_date)
+
+    def test_parse_timestamp_fails_for_invalid_timestamps(self):
+        with self.assertRaises(ValueError):
+            parse_timestamp('invalid')
+
+    def test_parse_timestamp_parses_millisecond_timestamps(self):
+        now = timezone.now().replace(microsecond=0)
+        fifty_years_ago = now - relativedelta(year=50)
+        fifty_years_from_now = now + relativedelta(year=50)
+
+        self.verify_timestamp(now)
+        self.verify_timestamp(fifty_years_ago)
+        self.verify_timestamp(fifty_years_from_now)
+
+    def test_parse_timestamp_parses_microsecond_timestamps(self):
+        now = timezone.now().replace(microsecond=0)
+        fifty_years_ago = now - relativedelta(year=50)
+        fifty_years_from_now = now + relativedelta(year=50)
+
+        self.verify_timestamp(now, 1000)
+        self.verify_timestamp(fifty_years_ago, 1000)
+        self.verify_timestamp(fifty_years_from_now, 1000)
+
+    def test_parse_timestamp_parses_nanosecond_timestamps(self):
+        now = timezone.now().replace(microsecond=0)
+        fifty_years_ago = now - relativedelta(year=50)
+        fifty_years_from_now = now + relativedelta(year=50)
+
+        self.verify_timestamp(now, 1000000)
+        self.verify_timestamp(fifty_years_ago, 1000000)
+        self.verify_timestamp(fifty_years_from_now, 1000000)
+
+    def test_parse_timestamp_fails_for_out_of_range_timestamp(self):
+        now = timezone.now().replace(microsecond=0)
+
+        with self.assertRaises(ValueError):
+            self.verify_timestamp(now, 1000000000)

bookmarks/utils.py

@@ -58,3 +58,40 @@ def humanize_relative_date(value: datetime, now: Optional[datetime] = None):
             return 'Yesterday'
         else:
             return weekday_names[value.isoweekday()]
+
+
+def parse_timestamp(value: str):
+    """
+    Parses a string timestamp into a datetime value
+    First tries to parse the timestamp as milliseconds.
+    If that fails with an error indicating that the timestamp exceeds the maximum,
+    it tries to parse the timestamp as microseconds, and then as nanoseconds
+    :param value:
+    :return:
+    """
+    try:
+        timestamp = int(value)
+    except ValueError:
+        raise ValueError(f'{value} is not a valid timestamp')
+
+    try:
+        return datetime.utcfromtimestamp(timestamp).astimezone()
+    except (OverflowError, ValueError, OSError):
+        pass
+
+    # Value exceeds the max. allowed timestamp
+    # Try parsing as microseconds
+    try:
+        return datetime.utcfromtimestamp(timestamp / 1000).astimezone()
+    except (OverflowError, ValueError, OSError):
+        pass
+
+    # Value exceeds the max. allowed timestamp
+    # Try parsing as nanoseconds
+    try:
+        return datetime.utcfromtimestamp(timestamp / 1000000).astimezone()
+    except (OverflowError, ValueError, OSError):
+        pass
+
+    # Timestamp is out of range
+    raise ValueError(f'{value} exceeds maximum value for a timestamp')

bookmarks/views/settings.py

@@ -14,6 +14,12 @@ from bookmarks.services import importer
 
 logger = logging.getLogger(__name__)
 
+try:
+    with open("version.txt", "r") as f:
+        app_version = f.read().strip("\n")
+except Exception as exc: 
+    logging.exception(exc)
+    pass
 
 @login_required
 def general(request):
@@ -30,6 +36,7 @@ def general(request):
         'form': form,
         'import_success_message': import_success_message,
         'import_errors_message': import_errors_message,
+        'app_version': app_version
     })
 
 

package.json

@@ -1,6 +1,6 @@
 {
   "name": "linkding",
-  "version": "1.6.5",
+  "version": "1.7.2",
   "description": "",
   "main": "index.js",
   "scripts": {

requirements.prod.txt

@@ -1,20 +1,22 @@
+asgiref==3.4.1
 beautifulsoup4==4.7.1
 certifi==2019.6.16
-chardet==3.0.4
+charset-normalizer==2.0.4
 confusable-homoglyphs==3.2.0
-Django==2.2.20
+Django==3.2.6
 django-generate-secret-key==1.0.2
-django-picklefield==2.0
-django-registration==3.1.2
-django-sass-processor==0.7.3
-django-widget-tweaks==1.4.5
-djangorestframework==3.11.2
+django-picklefield==3.0.1
+django-registration==3.2
+django-sass-processor==1.0.1
+django-widget-tweaks==1.4.8
+djangorestframework==3.12.4
 idna==2.8
 pyparsing==2.4.7
 python-dateutil==2.8.1
-pytz==2019.1
-requests==2.22.0
+pytz==2021.1
+requests==2.26.0
 soupsieve==1.9.2
-sqlparse==0.3.0
-urllib3==1.25.8
+sqlparse==0.4.1
+typing-extensions==3.10.0.0
+urllib3==1.26.6
 uWSGI==2.0.18

requirements.txt

@@ -1,27 +1,29 @@
+asgiref==3.4.1
 beautifulsoup4==4.7.1
 certifi==2019.6.16
-chardet==3.0.4
+charset-normalizer==2.0.4
 confusable-homoglyphs==3.2.0
 coverage==5.5
-Django==2.2.20
-django-appconf==1.0.3
-django-compressor==2.3
+Django==3.2.6
+django-appconf==1.0.4
+django-compressor==2.4.1
 django-debug-toolbar==3.2.1
 django-generate-secret-key==1.0.2
-django-picklefield==2.0
-django-registration==3.1.2
-django-sass-processor==0.7.3
-django-widget-tweaks==1.4.5
-djangorestframework==3.11.2
+django-picklefield==3.0.1
+django-registration==3.2
+django-sass-processor==1.0.1
+django-widget-tweaks==1.4.8
+djangorestframework==3.12.4
 idna==2.8
-libsass==0.19.2
+libsass==0.21.0
 pyparsing==2.4.7
 python-dateutil==2.8.1
-pytz==2019.1
+pytz==2021.1
 rcssmin==1.0.6
-requests==2.22.0
+requests==2.26.0
 rjsmin==1.1.0
-six==1.12.0
+six==1.16.0
 soupsieve==1.9.2
-sqlparse==0.3.0
-urllib3==1.25.8
+sqlparse==0.4.1
+typing-extensions==3.10.0.0
+urllib3==1.26.6

siteroot/settings/base.py

@@ -73,6 +73,8 @@ TEMPLATES = [
     },
 ]
 
+DEFAULT_AUTO_FIELD = 'django.db.models.AutoField'
+
 WSGI_APPLICATION = 'siteroot.wsgi.application'
 
 # Database

version.txt

@@ -1 +1 @@
-1.6.5
+1.7.2